PlotEdit: Natural Language-Driven Accessible Chart Editing in PDFs via Multimodal LLM Agents

Franck Dernoncourt; Kanika Goswami; Puneet Mathur; Ryan Rossi

REVIEW 3 major objections 6 minor 12 references

PlotEdit: Natural Language-Driven Accessible Chart Editing in PDFs via Multimodal LLM Agents

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chart images in PDFs can be edited by natural-language instructions, without the source data or code, by decomposing the chart, editing its parts, and re-rendering it through coordinated multimodal agents.

desk verdict Sensible multi-agent chart-editing system, but the evaluation is too thin and partly metric-gamed to support the paper's headline numbers. read the letter →

arxiv 2501.11233 v1 pith:6EFKLK5H submitted 2025-01-20 cs.IR cs.CLcs.MA

Kanika Goswami , Puneet Mathur , Ryan Rossi , Franck Dernoncourt This is my paper · ORCID

classification cs.IRcs.CLcs.MA

keywords charteditingLLMagentsde-renderingmultimodalfeedbackself-reflectionPDFaccessibilitynaturallanguagevisualfidelity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PlotEdit claims that charts stored as images inside PDFs or scans can be edited by natural-language instructions even when the original data table, style definitions, and source code are gone. The paper builds a five-agent pipeline that first de-renders the chart into a data table, visual attributes, and Python plotting code, then decomposes the user's request into individual edits, applies those edits to the appropriate component, and re-renders the chart. Three self-reflection loops, based on code validation, image-region comparison, and data-summary comparison, catch de-rendering errors before editing. On a benchmark of chart edits, the pipeline is reported to outperform direct in-context prompting and prior chart-editing models across style, layout, format, and data edits. The motivation is practical: making chart editing accessible to visually impaired readers and to non-experts who do not have the source files or specialized software.

What carries the argument

The load-bearing mechanism is a de-render, edit, re-render loop. Three retrieval agents build the editable representation: Chart2Table extracts the data table through chain-of-thought prompting, Chart2Vision records colors, fonts, markers, and legend mappings in JSON, and Chart2Code generates Python visualization code. Three additional feedback agents then refine that representation before any edit is made: code feedback parses and executes the code to catch errors, visual feedback compares original and replotted images in local regions using multi-scale structural similarity, and numeric feedback compares chart summaries and plot-to-table statistics to catch data mismatches. The Instruction Decomposition Agent turns the user request into a step sequence, the Multimodal Editing Agent applies those steps to the table, style JSON, or code, and a final perceptual-fidelity check confirms that unchanged regions stay untouched.

What would settle it

Take a set of real scanned or low-resolution chart images whose original data and rendering code are known, run PlotEdit's de-rendering agents on them, and compare the extracted table and code against the ground truth; if the extracted representation cannot reproduce the original chart within the allowed feedback rounds, then the edit quality predicted by the framework collapses on the precise input type the paper targets.

Watch

Extended reading notes

Core claim

Put in the paper's own terms, the central claim is that the bottleneck in language-driven chart editing is not the model's ability to apply an edit but its ability to de-render the chart image faithfully beforehand. PlotEdit therefore converts the chart into three editable artifacts, a data table, a JSON of visual attributes, and executable plotting code, and only then lets an editing agent modify them. The claimed payoff is that this decomposition, guided by multimodal feedback, produces edited charts that preserve the original's visual integrity while carrying out the requested change, and does so more reliably than asking a single model to reason directly from pixels. The paper further claims that the feedback loop is what separates the approach from weaker baselines, since it turns de-rendering mistakes into correctable errors instead of silent hallucinations.

Load-bearing premise

The load-bearing premise is that a vision-language model can reliably decompose a chart image into a correct data table, style attributes, and code; if that de-rendering step is inaccurate, every edit made downstream inherits the error.

Editorial extensions

If this is right

Charts in PDFs and scans become editable without recovering the original spreadsheet or script, removing the main practical barrier to editing legacy documents.
De-rendering errors that would otherwise corrupt an edited chart become visible and correctable, because the same image is re-rendered and compared with the input.
The data table produced during de-rendering can serve as an accessibility artifact, such as textual or machine-readable chart content.
A user who can describe a change in plain language can apply style, layout, format, and data edits without learning plotting libraries or design tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the loop transfers to other image-to-code tasks, the same de-render, verify, edit, re-render scaffold could apply to diagrams, maps, or screenshots, since the three feedback signals are not chart-specific.
Editorial inference: the reported advantage is tied to the current de-rendering capability of the underlying multimodal models; the durable contribution may be the self-reflection architecture, whose value should grow as the base models improve.
Editorial inference: a direct test of the accessibility claim would measure whether screen-reader output or novice completion time actually improves; the paper's evidence is image-similarity and data-fidelity metrics rather than these downstream outcomes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

PlotEdit is a competent engineering contribution to a narrow but real problem: editing charts that only exist as images in PDFs. The new part is the orchestration—five LLM agents that de-render a chart into table/style/code, decompose the user request, edit, and then iteratively correct themselves using code, numeric, and visual feedback. The ablation (PlotEdit w/o MFA) suggests the feedback loop is doing useful work. That is a legitimate contribution to the chart-editing subfield, and the accessibility motivation is genuine. The paper also targets the right failure mode: LLM prompting fails at de-rendering, so self-correction is a sensible instinct. The references are the right ones—ChartReformer, ChartLlama, DePlot—and I don't see a citation problem.

But the evidence as printed does not support the headline. The evaluation is a single table with no error bars, no statistical tests, no dataset splits, no prompt or code release, and ChartLlama's V AES/RMS values are blank. The abstract's 9–14% improvement over strong baselines is not in the table: the largest visible relative gain over in-context learning is about 8% (layout SSIM), with style/format/data gains mostly in the 1.5–4% range. That is a real overstatement.

The deeper problem is metric design. The Visual Feedback and Perceptual Fidelity Feedback agents use MS-SSIM/SSIM to keep the edited chart close to the original, and the headline evaluation metric is SSIM against the original. A system that aggressively preserves the input can therefore inflate SSIM without actually implementing the requested edit. The paper reports no semantic verification—no check that a data range was filtered or a color changed. This does not sink the paper, because the comparison against external baselines on the same metric is still informative, but it means the gains are not yet a measure of edit correctness. The whole pipeline also inherits GPT-4V de-rendering errors; the feedback loops mitigate this, but the paper doesn't show how often de-rendering fails.

Who should read this: people working on chart understanding/editing, and anyone designing evaluation for LLM-agent pipelines. It is a good case study in why pixel-similarity metrics are not enough. I would not cite it in my own work until the eval is tightened. For peer review: yes, it deserves referee time, but only with a request for code/data, confidence intervals, and a semantic edit-correctness check—and a corrected abstract.

Referee Report

3 major / 6 minor

Summary. The paper proposes PlotEdit, a five-LLM-agent pipeline for natural-language chart editing from PDF or scanned chart images: Chart2Table extracts the data table, Chart2Vision extracts visual attributes, Chart2Code retrieves rendering code, the Instruction Decomposition Agent splits user requests into executable steps, and the Multimodal Editing Agent applies data, style, and code edits. Three feedback mechanisms (code, visual, and numeric) refine the de-rendering agents, and a perceptual-fidelity feedback loop constrains the editing agent to preserve unchanged chart regions. The system is evaluated on the ChartCraft dataset against ChartLlama, ChartReformer, and in-context learning using SSIM, V AES, and RMS, with an ablation that removes multimodal feedback. The reported table shows PlotEdit achieving the best scores overall, with gains over the strongest baseline that are substantially smaller than the 9-14% claimed in the abstract.

Significance. If the quantitative claim were fully supported, PlotEdit would be a useful contribution to chart editing in PDFs and scanned documents, and its modular decomposition of chart de-rendering into table, style, and code components is a sensible design. The paper has concrete engineering strengths: the code feedback agent combines static AST checks with dynamic execution, the retrieval agents are given explicit modality-specific outputs, and the ablation (PlotEdit w/o MFA) provides some evidence about which components matter. However, the central claim of 9-14% improvement over strong baselines is not visible in Table 1, the metrics overlap with the optimization objectives used in the feedback loops, and the evaluation lacks the details needed to verify the comparison. These issues are load-bearing because the paper's main contribution is the reported performance.

major comments (3)

[Abstract and §3, Table 1] The headline claim that PlotEdit outperforms strong baselines by 9-14% is not supported by Table 1. The largest per-category gain over in-context learning (the strongest baseline) is 8.0 points in layout SSIM (91.3 vs. 84.5); style, format, and data gains are roughly 1.5-4.4 points across metrics, and overall SSIM/VAES/RMS gains are 2.2/2.7/1.4 points. Please reconcile the abstract with the table, or report the calculation that yields 9-14%. This is load-bearing because the abstract's quantitative promise is the paper's main result.
[§2, Visual Feedback and Perceptual Fidelity Feedback; §3, metrics] The evaluation is partially circular with the optimization objective. Visual Feedback iteratively adjusts Chart2Vision using MS-SSIM between the original and replot, and Perceptual Fidelity Feedback revises edits using region-specific SSIM against the original chart; SSIM against the original is then the headline evaluation metric. A system that minimizes SSIM deviation without implementing the requested change could therefore score well. The paper does not report any semantic verification that edits are actually realized (e.g., that a data range was filtered, a color changed, or a format converted according to the instruction). Please add an edit-success rate or human evaluation per edit type, or a control condition with a no-op instruction.
[§3, Table 1] The quantitative comparison is incomplete and not independently reproducible: ChartLlama's V AES and RMS cells are missing, no error bars or statistical tests are reported, the number and composition of ChartCraft test instances is not given, prompt templates and agent version numbers (e.g., GPT-4V/4o dates) are not specified, and ChartCraft itself is not in the reference list. These omissions block verification of the claim that PlotEdit significantly outperforms all baselines, especially for ChartLlama, where only SSIM is available.

minor comments (6)

[§2] The phrase "comprises of" is nonstandard; "comprises" or "consists of" would be correct.
[Figure 1] The figure contains a great deal of textual detail that is illegible at the printed size; a higher-resolution version or a simplified schematic would help readers follow the agent flow.
[§3] The metric definitions are too terse: RMS is only named as "Relative Mapping Similarity" and V AES is only described as "style precision," with no formulas or alignment details; this makes the table hard to interpret.
[§3] The phrase "significantly outperforms" is used without any statistical test or error bar; please qualify it or supply uncertainty estimates.
[§2] The feedback loops are described as running "until satisfactory results or exhaustion of max trials," but the maximum number of trials and the stopping criterion are never specified.
[§3] The application claim of "enhancing accessibility for visually challenged users" is not evaluated with any user study or accessibility-specific metric; it is presented as a consequence of color/contrast editing but no evidence is given.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PlotEdit is an empirical multi-agent pipeline with no derivation chain that reduces to its own inputs.

full rationale

PlotEdit makes no formal derivation claim; it is an empirical system built from LLM agents and evaluated on the ChartCraft benchmark. The paper contains no equations in which an output is defined in terms of the target quantity, no fitted parameter that is later renamed as a prediction, and no load-bearing self-citation. The retrieval agents (Chart2Table, Chart2Vision, Chart2Code) and feedback agents (Code, Visual, Numeric) are system components, not fitted parameters. The use of MS-SSIM in Visual Feedback and region-specific SSIM in Perceptual Fidelity Feedback is an optimization/refinement mechanism, while the reported SSIM is an evaluation metric following the prior ChartReformer protocol; both use structural similarity, but this is a shared metric family, not a definitional equivalence between the framework's objective and the evaluation outcome. The skeptical concern that the system may maximize SSIM without semantically implementing the requested edit is a question of metric validity and experimental verification, not circularity. Likewise, the discrepancy between the abstract's 9-14% improvement and the visible Table 1 margins is a reporting/consistency issue, not a circular-reasoning issue. The comparison against external baselines (ChartLlama, ChartReformer, in-context learning) on a fixed dataset is self-contained and falsifiable, so the central empirical claim is not forced by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on assumptions about the reliability of GPT-4V for chart de-rendering, the validity of DePlot and MS-SSIM as feedback signals, and the representativeness of the ChartCraft dataset. No explicit free parameters are reported, and no speculative entities are introduced.

assumptions (4)

domain assumption Charts in the evaluation set can be faithfully reconstructed from extracted data, style attributes, and Python code.
The entire de-rendering and re-plotting pipeline assumes a chart is a lossless render of tabular data and code, which is true for ChartCraft's synthetic charts but not guaranteed for real scanned PDFs.
domain assumption GPT-4V can accurately extract data tables, visual attributes, and Python code from chart images.
Retrieval agents rely on GPT-4V; any extraction error propagates through editing and affects final quality.
domain assumption DePlot provides reliable local quantitative measures (averages, extrema, discontinuities) for comparing original and replotted charts.
Numeric feedback uses DePlot as ground truth for data consistency.
domain assumption MS-SSIM and GPT-4V region feedback are valid indicators of visual fidelity.
Visual feedback and perceptual fidelity feedback trust these signals to guide refinement without independent verification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PlotEdit: Natural Language-Driven Accessible Chart Editing in PDFs via Multimodal LLM Agents." pith.science (2026). https://pith.science/paper/6EFKLK5H

@misc{pith2026250111233,
  author       = {Pith},
  title        = {Pith review of: PlotEdit: Natural Language-Driven Accessible Chart Editing in PDFs via Multimodal LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EFKLK5H}},
  note         = {Machine review of arXiv:2501.11233}
}

read the original abstract

Chart visualizations, while essential for data interpretation and communication, are predominantly accessible only as images in PDFs, lacking source data tables and stylistic information. To enable effective editing of charts in PDFs or digital scans, we present PlotEdit, a novel multi-agent framework for natural language-driven end-to-end chart image editing via self-reflective LLM agents. PlotEdit orchestrates five LLM agents: (1) Chart2Table for data table extraction, (2) Chart2Vision for style attribute identification, (3) Chart2Code for retrieving rendering code, (4) Instruction Decomposition Agent for parsing user requests into executable steps, and (5) Multimodal Editing Agent for implementing nuanced chart component modifications - all coordinated through multimodal feedback to maintain visual fidelity. PlotEdit outperforms existing baselines on the ChartCraft dataset across style, layout, format, and data-centric edits, enhancing accessibility for visually challenged users and improving novice productivity.

Figures

Figures reproduced from arXiv: 2501.11233 by the authors.

**Figure 1.** PlotEdit accurately edits chart images as per user requests by orchestrating LLM agents: (1) Chart2Table for data table extraction, (2) Chart2Vision for style attribute identification, (3) Chart2Code for retrieving rendering code, (4) Instruction Decomposition Agent for parsing user requests into executable steps, and (5) Multimodal Editing Agent for implementing nuanced chart component modifications coordinated thr… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 8 canonical work pages

[1]

ArXiv abs/2311.16483 (2023), https://api.semanticscholar.org/CorpusID:265466206

Han, Y., Zhang, C.X., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., Zhang, H.: Chartllama: A multimodal llm for chart understanding and generation. ArXiv abs/2311.16483 (2023), https://api.semanticscholar.org/CorpusID:265466206

arXiv 2023
[2]

ArXiv abs/2212.10505 (2022), https://api.semanticscholar.org/CorpusID:254877346

Liu, F., Eisenschlos, J.M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., Altun, Y.: Deplot: One-shot visual language reasoning by plot-to-table translation. ArXiv abs/2212.10505 (2022), https://api.semanticscholar.org/CorpusID:254877346

arXiv 2022
[3]

IEEE Transactions on Visualization and Computer Graphics 27, 369--379 (2020), https://api.semanticscholar.org/CorpusID:221292836

Narechania, A., Srinivasan, A., Stasko, J.T.: Nl4dv: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics 27, 369--379 (2020), https://api.semanticscholar.org/CorpusID:221292836

work page 2020
[4]

IEEE Transactions on Visualization and Computer Graphics 23, 341--350 (2018), https://api.semanticscholar.org/CorpusID:206805969

Satyanarayan, A., Moritz, D., Wongsuphasawat, K., Heer, J.: Vega-lite: A grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics 23, 341--350 (2018), https://api.semanticscholar.org/CorpusID:206805969

work page 2018
[5]

In: Annual Meeting of the Association for Computational Linguistics (2020), https://api.semanticscholar.org/CorpusID:218611161

Shao, Y., Nakashole, N.: Chartdialogs: Plotting from natural language instructions. In: Annual Meeting of the Association for Computational Linguistics (2020), https://api.semanticscholar.org/CorpusID:218611161

work page 2020
[6]

IEEE Transactions on Visualization and Computer Graphics 24, 511--521 (2018), https://api.semanticscholar.org/CorpusID:2244239

Srinivasan, A., Stasko, J.T.: Orko: Facilitating multimodal interaction for visual exploration and analysis of networks. IEEE Transactions on Visualization and Computer Graphics 24, 511--521 (2018), https://api.semanticscholar.org/CorpusID:2244239

work page 2018
[7]

IEEE transactions on image processing 13(4), 600--612 (2004)

Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600--612 (2004)

work page 2004
[8]

In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003

Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. vol. 2, pp. 1398--1402. Ieee (2003)

work page 2003

Show all 12 references

[9]

Advances in neural information processing systems 35, 24824--24837 (2022)

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824--24837 (2022)

2022
[10]

In: International Conference on Document Analysis and Recognition

Yan, P., Bhosale, M., Lal, J., Adhikari, B., Doermann, D.: Chartreformer: Natural language-driven chart image editing. In: International Conference on Document Analysis and Recognition. pp. 453--469. Springer (2024)

2024
[11]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[12]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

ArXiv abs/2311.16483 (2023), https://api.semanticscholar.org/CorpusID:265466206

Han, Y., Zhang, C.X., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., Zhang, H.: Chartllama: A multimodal llm for chart understanding and generation. ArXiv abs/2311.16483 (2023), https://api.semanticscholar.org/CorpusID:265466206

arXiv 2023

[2] [2]

ArXiv abs/2212.10505 (2022), https://api.semanticscholar.org/CorpusID:254877346

Liu, F., Eisenschlos, J.M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., Altun, Y.: Deplot: One-shot visual language reasoning by plot-to-table translation. ArXiv abs/2212.10505 (2022), https://api.semanticscholar.org/CorpusID:254877346

arXiv 2022

[3] [3]

IEEE Transactions on Visualization and Computer Graphics 27, 369--379 (2020), https://api.semanticscholar.org/CorpusID:221292836

Narechania, A., Srinivasan, A., Stasko, J.T.: Nl4dv: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics 27, 369--379 (2020), https://api.semanticscholar.org/CorpusID:221292836

work page 2020

[4] [4]

IEEE Transactions on Visualization and Computer Graphics 23, 341--350 (2018), https://api.semanticscholar.org/CorpusID:206805969

Satyanarayan, A., Moritz, D., Wongsuphasawat, K., Heer, J.: Vega-lite: A grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics 23, 341--350 (2018), https://api.semanticscholar.org/CorpusID:206805969

work page 2018

[5] [5]

In: Annual Meeting of the Association for Computational Linguistics (2020), https://api.semanticscholar.org/CorpusID:218611161

Shao, Y., Nakashole, N.: Chartdialogs: Plotting from natural language instructions. In: Annual Meeting of the Association for Computational Linguistics (2020), https://api.semanticscholar.org/CorpusID:218611161

work page 2020

[6] [6]

IEEE Transactions on Visualization and Computer Graphics 24, 511--521 (2018), https://api.semanticscholar.org/CorpusID:2244239

Srinivasan, A., Stasko, J.T.: Orko: Facilitating multimodal interaction for visual exploration and analysis of networks. IEEE Transactions on Visualization and Computer Graphics 24, 511--521 (2018), https://api.semanticscholar.org/CorpusID:2244239

work page 2018

[7] [7]

IEEE transactions on image processing 13(4), 600--612 (2004)

Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600--612 (2004)

work page 2004

[8] [8]

In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003

Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. vol. 2, pp. 1398--1402. Ieee (2003)

work page 2003