Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Most image editors can change chart labels but fail to update the linked bars, slices, and geometry that the labels describe.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:47 UTC pith:G5T2Y4EF

load-bearing objection Clean diagnostic for a real failure mode: most editors still treat chart numbers as free text, not as geometry drivers; only two frontier systems show emerging VLCE. the 3 major comments →

arxiv 2607.10301 v1 pith:G5T2Y4EF submitted 2026-07-11 cs.CV cs.CL

ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing

Jiakang Yu , Yixuan Chai , Tianci Wang , Rihui Jin , Guangkai Xu , Hongtao Deng , Xun Zhu , Wang Gao
show 2 more authors
Xinrun Guo Haipang Wu
This is my paper
classification cs.CV cs.CL
keywords chart editingvisuo-logical cascading editingVLCEimage editing benchmarktext-to-geometry synchronizationvision-language modelsstructured document understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Instruction-based image editors work on natural photos, yet statistical charts demand a harder skill: when a number changes, the drawn geometry must change with it. The authors formalize that requirement as Visuo-Logical Cascading Editing (VLCE) and release ChartSync, an expert-checked set of 870 original-instruction-ground-truth triplets spanning nine chart types, including 235 cases that force text-to-geometry synchronization. Ground truth is produced by editing the original plotting code and re-rendering, so the correct geometry is deterministic rather than hand-drawn. A two-tier scoreboard pairs ordinary image metrics with a vision-language judge that separately rates textual success, geometric consistency, and background preservation. Across fourteen editors and one chart-to-code pipeline, most models still treat the task as local text replacement; only two frontier proprietary systems show clear cascading ability, and even they leave residual isolation and background errors. The gap isolates three meta-abilities future multimodal systems will need: grounding text to marks, reasoning over data-driven shape changes, and editing without bleeding into the rest of the figure.

Core claim

On ChartSync, reliable text-to-geometry cascading is rare. Textual edit success often remains moderate to high while Visuo-Logical Consistency collapses for open-source and many proprietary editors (for example one strong open model drops from about 62 textual success to under 14 geometric consistency). Only two frontier proprietary models reach strong VLCS scores (roughly 75 and 84); residual failures for those systems cluster in semantic isolation and background corruption. A code-mediated reconstruct-edit-render path can rewrite labels yet still loses geometric and layout fidelity when the original source is unavailable.

What carries the argument

Visuo-Logical Cascading Editing (VLCE) plus the ChartSync VLCE subset and VLCS score: a textual value change must induce the matching geometric deformation (bar height, pie angle, etc.) while non-target regions stay intact, measured on 235 geometry-coupled instances with deterministic code-rendered ground truth.

Load-bearing premise

The ranking of cascading skill rests on a single proprietary vision-language judge for geometric consistency and background fidelity, even though that judge was checked against a few hundred human ratings.

What would settle it

Re-score the same 235 VLCE predictions with a different-family human panel or multi-model judge ensemble; if the large TESR-to-VLCS gaps reverse or the two frontier models lose their lead, the central capability claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pixel-space chart editors must be tested on value-to-geometry coupling, not only OCR or global SSIM.
  • Code-only pipelines are not a free fix when source plots are unavailable, because reconstruction loses layout and geometry.
  • Future architectures need three stacked meta-abilities: perception grounding, data-driven geometric reasoning, and artifact-free isolation.
  • Public release of the 870 triplets and rendering pipeline gives a fixed diagnostic for measuring progress on structured document editing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training regimes that only reward literal text replacement will keep producing the observed TESR–VLCS cliff until geometric coupling is an explicit objective.
  • The same cascading test could be ported to other rigid diagrams (timelines, flowcharts, scientific schematics) where labels control shape.
  • Residual background bleeding even in strong models suggests that mask-free diffusion editors still lack reliable region isolation for dense symbolic graphics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper formalizes Visuo-Logical Cascading Editing (VLCE) for statistical charts—edits where a textual value change must trigger synchronized geometric deformation—and introduces ChartSync, an expert-validated benchmark of 870 image–instruction–GT triplets across 9 chart types and 4 task types, of which 235 are geometry-coupled VLCE instances. Ground truth is produced by a three-stage programmatic pipeline (instruction synthesis, code-driven re-rendering from ChartMimic sources, unanimous three-expert QA retaining 870/935 candidates). Evaluation uses a two-tier protocol (OCR F1, SSIM plus VLM-as-a-Judge TESR/VLCS/BFS) on 14 image editors and one chart-to-code pipeline. Main empirical claim: most open-source and several proprietary models show large TESR–VLCS drops (e.g., Qwen-Image-Edit-2511: 61.81 vs 13.83), while only Nano Banana Pro and GPT-Image-2 exhibit strong emerging text-to-geometry synchronization (VLCS 83.71 and 74.47), with residual failures in semantic isolation and background corruption. Failure modes are distilled into three hierarchical meta-abilities; data and code are released.

Significance. If the reported ranking holds, ChartSync is a useful diagnostic benchmark that isolates value-to-geometry synchronization—an ability prior chart-editing suites (ChartEdit, ChartM3, ChartE³, ChartEditVista) do not score as a first-class target (Table 1). Strengths that raise the contribution above a routine leaderboard paper include: deterministic code-rendered GT with expert unanimous QA (Appendix A, Table 4), public release of dataset and construction code, a calibrated two-tier metric suite with blind human correlation (ICC 0.87; VLCS Pearson r=0.892 on 200 samples, Table 3), and a code-mediated baseline that shows reconstruction loss is not a free solution. The TESR–VLCS gap and the qualitative cases (Figs. 13–16) give a concrete target for multimodal editing architectures.

major comments (3)
  1. [§4.2, Table 2, Table 3] §4.2 and Table 2: The central ranking claim—that only Nano Banana Pro and GPT-Image-2 show strong VLCE—rests primarily on Gemini-judged VLCS over the 235 VLCE instances. Table 3 reports strong overall human correlation (VLCS r=0.892, n=200), but the paper does not break out calibration statistics restricted to the VLCE subset or report inter-judge agreement for VLCS specifically. Given that Limitations already flags single-family judge risk, a modest expansion (e.g., second-family judge or human VLCS on a stratified VLCE subsample with reported agreement) is needed so the TESR–VLCS gap ranking is not over-dependent on one evaluator family.
  2. [§5.2, Figure 6] §5.2 and Fig. 6: The three hierarchical meta-abilities (foundation perception, synchronization/reasoning, high-fidelity generation) are presented as general architectural guidance, yet the error taxonomy is derived only from Qwen-Image-Edit-2511 vs Nano Banana Pro. Limitations item (4) acknowledges this, but the main text still generalizes from a two-model multi-label breakdown. Either restrict the meta-ability claims to “representative failure modes of the top open-source and top proprietary systems,” or add a lighter taxonomy pass over a few additional models so the hierarchy is not load-bearing on an n=2 comparison.
  3. [§3.4.2, Table 2] §3.4.2 / Overall Score definition: Overall Score is the equal average of OCR F1, SSIM, TESR, VLCS, and BFS, while VLCS is defined only on the 235 VLCE samples and is not assigned on text-only items. The paper notes this is a “metric-level summary,” but Table 2 rankings and the abstract’s capability narrative mix full-set and VLCE-only signals. Please report a VLCE-subset Overall (or primary VLCS/TESR/BFS on the 235) alongside the full-set summary so the cascading-reasoning claim is not diluted or inflated by the 635 text-only triplets.
minor comments (5)
  1. [§4.1, Table 2] Model identifiers such as “Nano Banana Pro” (footnote 2: gemini-3-pro-image) and “GPT-Image-2” should be stated once with canonical API names in the main text of §4.1 for reproducibility; the footnotes alone are easy to miss.
  2. [§5.1, Figure 5] Figure 5 compares only two models across categories/tasks; a compact appendix table of VLCS by chart family for all models (or at least all proprietary + top open-source) would make the “fluctuations on bar/pie/multidiff” claim checkable.
  3. [§3.1, Eq. (1)] Eq. (1) writes ΔG = S(ΔV) without specifying the type of S; a one-sentence note that S is the deterministic renderer-induced map (not a learned operator) would avoid misreading S as a model component.
  4. [§3.4.2, Appendix D] Appendix D judge prompt omits 0.75 for VLCS by design; a short pointer in §3.4.2 to that design choice (already in the appendix) would help readers who only skim the main metrics section.
  5. [Table 1] Table 1 “Value-to-Geometry Sync. / Sync. Metric” columns correctly distinguish ChartSync; ensure ChartE³ and ChartEditVista citations remain accurate if those arXiv versions evolve before camera-ready.

Circularity Check

0 steps flagged

No significant circularity: ChartSync is an empirical benchmark with programmatically rendered, expert-validated GT and independent model evaluation.

full rationale

This paper does not claim a first-principles derivation or a fitted-parameter prediction that reduces to its inputs. Ground-truth images are produced by editing original ChartMimic plotting code under a logic validator and unanimous expert QA (Appendix A, Table 4), then re-rendering—so visuo-logical coupling is enforced by construction of the benchmark, not by scoring models against their own outputs. Model rankings (Table 2) rest on objective OCR F1/SSIM plus a Gemini VLM judge that is separately calibrated against blind human ratings (Table 3: VLCS r=0.892, ICC 0.87). Construction agents (GPT-5.4) and the judge (Gemini-3.1-Pro) are different systems; residual single-family judge risk is already flagged in Limitations and does not make TESR/VLCS scores tautological. There is no self-definitional equation, no fitted input renamed as prediction, no load-bearing uniqueness theorem imported from overlapping authors, and no renaming of a known result as a derived law. Empty steps is the correct outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

As a benchmark paper, load-bearing content is mostly operational definitions and evaluation design choices rather than physical free parameters. The central claim depends on treating charts as value–geometry dependency graphs, on the discrete judge rubrics, and on the invented VLCE/VLCS constructs. No continuous parameters are fitted to force a theoretical curve; the free choices are scoring grids and construction filters.

free parameters (4)
  • TESR discrete score grid = {0.0,0.2,0.4,0.6,0.8,1.0}
    Author-chosen six-level set {0.0,0.2,...,1.0} used for textual edit success; not derived from an external standard.
  • VLCS discrete score grid = {0.0,0.25,0.5,1.0}
    Author-chosen four-level set with intentional omission of 0.75; directly drives the main cascading-capability claim.
  • BFS discrete score grid = {0.0,0.25,0.5,0.75,1.0}
    Author-chosen five-level background fidelity scale used in VLM Avg and Overall.
  • Overall Score equal-weight average = equal mean of 5 metric means
    Equal average of OCR F1, SSIM, TESR, VLCS, BFS is a hand-chosen aggregation, not sample-level averaging; ranking sensitivity depends on this choice.
axioms (5)
  • domain assumption A structured chart can be modeled as C=(V,G,S) with geometric change ΔG=S(ΔV).
    Section 3.1 task definition; treats value–geometry coupling as the right abstraction for evaluation.
  • domain assumption Programmatic code edits plus re-rendering yield unique, correct ground-truth images for cascading edits.
    Phase 2–3 construction; underpins all supervised metrics against I_gt.
  • ad hoc to paper Unanimous acceptance by three Ph.D. AI experts is sufficient quality control for the 870 retained triplets.
    Appendix A QA protocol; no external inter-lab replication of the accept/reject decisions.
  • domain assumption A proprietary VLM judge can score multimodal comprehension dimensions that pixel metrics miss.
    Section 3.4.2 and human correlation Table 3; central to VLCS/TESR/BFS claims.
  • standard math Character-level OCR F1 and SSIM are adequate low-level fidelity proxies for chart edits.
    Standard metrics adapted in §3.4.1; used as objective tier alongside the judge.
invented entities (3)
  • Visuo-Logical Cascading Editing (VLCE) no independent evidence
    purpose: Name and formalize the requirement that text value edits must induce synchronized geometric updates and non-target preservation.
    Core task formalization in §1 and §3.1; evaluation claims are defined relative to this construct.
  • ChartSync benchmark (870 triplets, VLCS metric) independent evidence
    purpose: Provide deterministic image-only test cases and a dedicated geometric synchronization score.
    Primary artifact; VLCS is new relative to cited chart-editing benchmarks that do not isolate value-to-geometry sync.
  • Three hierarchical meta-abilities (foundation perception, synchronization/reasoning, high-fidelity generation) no independent evidence
    purpose: Organize observed failure modes into guidance for future architectures.
    §5.2 post-hoc taxonomy from two representative models; useful framing but not independently measured constructs.

pith-pipeline@v1.1.0-grok45 · 23748 in / 3494 out tokens · 42884 ms · 2026-07-14T12:47:01.291807+00:00 · methodology

0 comments
read the original abstract

Generative image editing models struggle with structured statistical charts when data modifications require geometric synchronization. We formalize this task as Visuo-Logical Cascading Editing (VLCE). However, existing methods remain confined to localized text substitutions and struggle with dependency-aware cascading updates. To systematically evaluate this capability, we introduce ChartSync, an expert-validated benchmark constructed via a programmatic rendering pipeline that guarantees deterministic visuo-logical coupling for the ground truth. ChartSync comprises 870 triplets across 9 chart categories and 4 task types, including 235 geometry-coupled VLCE instances that specifically test cascading text-to-geometry synchronization. We further evaluate these instances via a two-tier framework combining objective visual metrics with a vision-language model judge paradigm to assess low-level fidelity alongside multimodal comprehension and reasoning. Evaluating 14 image editing models and one code-mediated pipeline reveals a nuanced capability gap: most open-source models suffer severe drops in geometric synchronization, while only two frontier proprietary models show emerging VLCE capability, with their residual errors mainly involving semantic isolation and background corruption. Our detailed error analysis deconstructs these failure paradigms to identify core meta-abilities for guiding future multimodal architectures. The ChartSync dataset and code are publicly released at https://github.com/kaka-yjk/ChartSyncCodebase.

Figures

Figures reproduced from arXiv: 2607.10301 by Guangkai Xu, Haipang Wu, Hongtao Deng, Jiakang Yu, Rihui Jin, Tianci Wang, Wang Gao, Xinrun Guo, Xun Zhu, Yixuan Chai.

Figure 1
Figure 1. Figure 1: An illustration of the VLCE task. While tradi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An overview of ChartSync, consisting of a dataset, a VLCE task formulation, and an evaluation framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The three-stage data construction pipeline of ChartSync. Phase 1 generates and evaluates initial editing [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Data distribution of chart categories and task [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison between Nano Banana Pro and Qwen-Image-Edit-2511. The left panel illustrates [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of primary error occurrences [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The prompt for the Instruction Proposer. It guides the model to identify semantic key-value pairs and [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The prompt for Instruction Evaluator. It serves as a quality gate to verify the format, target alignment, and [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The prompt for Code Generator. It enforces visuo-logical propagation rules across diverse chart families [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The prompt for Logic Validator. It acts as an iterative verification mechanism to ensure that programmatic [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The GPT-5.4-based prompt used for OCR extraction in OCR F [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The prompt for VLM Judge. It defines the holistic evaluation criteria across three decoupled dimensions: [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Case 1: Qualitative comparison of generative chart editing given a complex multi-value instruction. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Case 2: Qualitative assessment under a single VLCE task. While Qwen-Image-Edit-2511 modifies [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Case 3: Qualitative example of a remaining semantic-isolation failure. Both models alter the target texts [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Case 4: Qualitative analysis under a complex cascading instruction. Both open-source and proprietary [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 2 linked inside Pith

  1. [1]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 18392–18402

    Instructpix2pix: Learning to follow image edit- ing instructions. InProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 18392–18402. Liangyu Chen, Yichen Xu, Jianzhe Ma, Yuqi Liu, Donglu Yang, Liang Zhang, Zihao Yue, Wenxuan Wang, and Qin Jin. 2026. Charteditor: A reinforce- ment learning framework for robust chart e...

    Pith/arXiv arXiv 2026

  2. [2]

    Agus Gunawan, Samuel Teodoro, Yun Chen, Soo Ye Kim, Jihyong Oh, and Munchurl Kim

    Texteditbench: Evaluating reasoning-aware text editing beyond rendering.arXiv preprint arXiv:2512.16270. Agus Gunawan, Samuel Teodoro, Yun Chen, Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2025. Omnitext: A training-free generalist for control- lable text-image manipulation.arXiv preprint arXiv:2510.24093. Zefeng He, Xiaoye Qu, Yafu Li, Tong Zhu, Siyuan Hua...

    arXiv 2025

  3. [3]

    Changyao Tian, Danni Yang, Guanzhou Chen, Erfei Cui, Zhaokai Wang, Yuchen Duan, Penghao Yin, Sitao Chen, Ganlin Yang, Mingxin Liu, and 1 others

    Firered-image-edit-1.0 technical report.arXiv preprint arXiv:2602.13344. Changyao Tian, Danni Yang, Guanzhou Chen, Erfei Cui, Zhaokai Wang, Yuchen Duan, Penghao Yin, Sitao Chen, Ganlin Yang, Mingxin Liu, and 1 others

    arXiv

  4. [4]

    Dianyi Wang, Ruihang Li, Feng Han, Chaofan Ma, Wei Song, Siyuan Wang, Yibin Wang, Yi Xin, Hongjian Liu, Zhixiong Zhang, and 1 others

    Internvl-u: Democratizing unified multimodal models for understanding, reasoning, generation and editing.arXiv preprint arXiv:2603.09877. Dianyi Wang, Ruihang Li, Feng Han, Chaofan Ma, Wei Song, Siyuan Wang, Yibin Wang, Yi Xin, Hongjian Liu, Zhixiong Zhang, and 1 others. 2026. Deepgen 1.0: A lightweight unified multimodal model for ad- vancing image gener...

    arXiv 2026

  5. [5]

    Psychology

    Imgedit: A unified image editing dataset and benchmark.Advances in Neural Information Pro- cessing Systems, 38. Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. 2023. Magicbrush: A manually annotated dataset for instruction-guided image editing.Ad- vances in Neural Information Processing Systems, 36:31428–31449. Bing Zhao, Chenfei Wu, Deqing Li, Hao...

    Pith/arXiv arXiv 2023

  6. [6]

    Key-Value

    Identify logical "Key-Value" pairs in the chart. - Case A (Inter-element): The Key is in one element (e.g., "Q3 Revenue for Tesla"), and the Value is in another (e.g., "450"). - Case B (Implicit Key in a Single Block): If a text block contains a clear value but no explicit key, select the target text as the'Value'and infer/generate a logical'Key'for it yourself

  7. [7]

    Axis", but rather represent the actual meaning of the axis (e.g.,

    Select ONE Key-Value pair to edit. The'Value'is the target to be modified ('old_value'). *IMPORTANT: Prioritize highly semantic and diverse text (e.g., specific amounts, dates, person names, addresses, specific metrics) over trivial elements. The'key'must NOT be a generic term without specific semantic meaning like "Axis", but rather represent the actual ...

  8. [8]

    (The'instruction'MUST contain the'key','old_value', and'new_value'.) [Critical Constraints]

    Generate a clear'instruction'for editing this value ('old_value'). (The'instruction'MUST contain the'key','old_value', and'new_value'.) [Critical Constraints]

  9. [9]

    Valid text element types in the chart include: `chart_title`,`subplot_title`,`axis_title`,`tick_label`,`legend_label`,`data_label`,`node_label`,`edge_label`

    ONLY select text elements in the chart. Valid text element types in the chart include: `chart_title`,`subplot_title`,`axis_title`,`tick_label`,`legend_label`,`data_label`,`node_label`,`edge_label`

  10. [10]

    The'new_value'MUST maintain exactly the same language and style as the'old_value', as well as the logical relationship corresponding to the key

  11. [11]

    You must precisely match the number of words (one word replaced by one word, no more or no less)

    Strict Alignment: The'new_value'MUST strictly maintain the exact same number of words as the'old_value'. You must precisely match the number of words (one word replaced by one word, no more or no less). Furthermore, the'new_value'MUST have roughly the same CHARACTER COUNT as the'old_value'to prevent layout breaking. Punctuation marks must also be replaced...

  12. [12]

    Location Precision: The'instruction'MUST rely on the'key'and'old_value'as the primary semantic anchors for a downstream model to locate the target text

  13. [13]

    category

    Proposing edits that create logical contradictions with the rest of the chart is perfectly ACCEPTABLE (e.g., an instruction requires changing the percentage label value of a pie chart without altering the corresponding pie chart slice size). These contradictions will be identified and fixed by downstream Agents. [Output Format] Please output strictly in J...

    1999

  14. [14]

    old_value

    "old_value" is a clear, localizable, and meaningful text target

  15. [15]

    category

    "category" is valid for charts and belongs to one of: chart_title, subplot_title, axis_title, tick_label, legend_label, data_label, node_label, edge_label

  16. [16]

    new_value

    "new_value" MUST have the exact same number of words and roughly the same character count as "old_value", matching its language and style

  17. [17]

    instruction

    "instruction" MUST contain the'key','old_value', and'new_value'

  18. [18]

    After replacement, the edited text looks natural in the chart

  19. [19]

    failure_type

    The sample has substantial training value. [Fail Conditions & Taxonomy] A proposal should FAIL if ANY of the following conditions are met. Assign the corresponding "failure_type": - "target_not_found": The "old_value" is vague, meaningless, or cannot be reliably localized in the chart. - "category_mismatch": The "category" is missing, invalid, or not in t...

  20. [20]

    logical inconsistencies

    DO NOT fail a proposal due to "logical inconsistencies" (e.g., broken arithmetic, timeline conflicts, or chart-text mismatches). We INTENTIONALLY allow these contradictions to test the self-correction capabilities of downstream agents

  21. [21]

    new_value

    DO NOT fail a proposal simply because the original image does not support the "new_value"

  22. [22]

    status":

    DO NOT treat this as a fact-checking or error-correcting task for the original document. [OUTPUT FORMAT] Please output strictly in JSON format. Return ONLY the raw JSON object. Do NOT wrap it in Markdown formatting (no```json ...```): { "status": "pass" or "fail" (strictly lowercase), "failure_type": "target_not_found", "category_mismatch", "format_mismat...

  23. [23]

    First identify the PRIMARY target required by the instruction

  24. [24]

    Then build a REQUIRED PROPAGATION SET: all code locations that must be updated to preserve semantic consistency

  25. [25]

    Keep edits minimal, but do not leave causal inconsistencies unresolved

  26. [26]

    Do not refactor unrelated code

  27. [27]

    [Propagation Workflow]

    Preserve executable Python syntax. [Propagation Workflow]

  28. [28]

    Locate the primary target

  29. [29]

    Find all same-entity/same-source references that must stay consistent

  30. [30]

    Apply only required propagation edits

  31. [31]

    thought":

    Re-check value relationships and chart geometry consistency. [Cascade Detection Rules: Target Category] - chart_title/subplot_title: if shared title text or shared title variable is reused across subplots, update all same-source occurrences. - axis_title: if x/y or multi-panel plots share same title variable/text convention, propagate to all same-source a...

  32. [32]

    edited_code

    "edited_code" MUST be the complete final Python code after editing

  33. [33]

    edited_code

    "edited_code" MUST be a valid JSON string with proper escaping for newlines and quotes

  34. [34]

    Do NOT output markdown code fences

  35. [35]

    [Source Code] {source_code} [Previous Failed Plan] {previous_plan_json} [Previous Validation Result] {previous_validation_json} [Feedback] {feedback_text} Return JSON only

    Do NOT wrap code in triple quotes. [Source Code] {source_code} [Previous Failed Plan] {previous_plan_json} [Previous Validation Result] {previous_validation_json} [Feedback] {feedback_text} Return JSON only. No markdown. Figure 9: The prompt for Code Generator. It enforces visuo-logical propagation rules across diverse chart families to ensure that code m...

  36. [36]

    Instruction alignment: edit matches the requested semantic target

  37. [37]

    Propagation-scope discipline: edits can span multiple locations, but only required propagation locations should be changed (no unrelated broad edits)

  38. [38]

    Old->new realization: replacement is actually reflected in code semantics, not only literal text

  39. [39]

    Semantic propagation consistency for target categories (chart_title/subplot_title/axis_title/tick_label/legend_label/data_label/node_label/edge_label)

  40. [40]

    Numeric and structural consistency: - sum/total/percentage relationships remain coherent, - chart geometry-driving data is synchronized with updated labels, - linked panels (multidiff/PIP) remain mutually consistent

  41. [41]

    Python safety: code appears syntactically valid and runnable

  42. [42]

    status":

    Logic consistency: no obvious internal contradiction after all propagated edits. [Evidence Guidance] When failing, provide concrete evidence snippets such as: - before/after sum or percentage relationships, - mismatches between label text and value arrays, - missing propagation locations for shared axis titles, ticks, or legends. [Failure Types] - instruc...

  43. [43]

    - Use Source vs GroundTruth to infer the REQUIRED text edit, then check ModelOutput vs GroundTruth

    Textual_Edit_Success (Score: 0.0, 0.2, 0.4, 0.6, 0.8, or 1.0): - STRICTLY evaluate ONLY the target TEXT mentioned in the Instruction. - Use Source vs GroundTruth to infer the REQUIRED text edit, then check ModelOutput vs GroundTruth. - Consider BOTH the text content (OCR-level correctness) AND its spatial properties (geometric center, alignment, scale/siz...

  44. [44]

    - Completely IGNORE the edited target text and its directly coupled geometric regions (e.g., the specific bar or pie slice being edited)

    Background_Fidelity (Score: 0.0, 0.25, 0.5, 0.75, or 1.0): - STRICTLY evaluate ONLY the UNTARGETED, background, and irrelevant parts of the chart, including rendering of non-target text. - Completely IGNORE the edited target text and its directly coupled geometric regions (e.g., the specific bar or pie slice being edited). - All other non-target structure...

  45. [45]

    Textual_Edit_Success

    Visuo_Logical_Consistency (Score: 0.0, 0.25, 0.5, or 1.0): (Provided if applicable) - STRICTLY evaluate ONLY the NON-TEXT geometric elements representing the data values. - Target Geometry: This includes bar/box height or length, line series position/shape, pie slice angle/area, error-bar length, radar point positions and filled area shape, scatter point ...