{"id":"1e1b617b-7f0e-4444-9585-6e63482acb98","arxiv_id":"2501.11233","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PlotEdit, a self-reflective multi-agent LLM pipeline, claims state-of-the-art results for natural-language chart editing on ChartCraft across style, layout, format, and data edits.","lead":"PlotEdit is a five-agent AI framework that edits charts in PDFs by first extracting data, style, and code from the chart image, then applying natural-language edit instructions. It reports higher style, layout, format, and data-edit scores than prior baselines on the ChartCraft dataset, but without released code or error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may reflect SSIM optimization rather than verified edit correctness; the abstract's 9–14% improvement is also not reproduced by Table 1.","rationale":"I read the paper as an engineering contribution that combines GPT-4V-based de-rendering, three feedback signals, and an editing agent. The strongest claim is the quantitative superiority over baselines on ChartCraft. The reader's weakest_assumption focuses on de-rendering accuracy, which is indeed a necessary condition: if Chart2Table, Chart2Vision, or Chart2Code produces wrong components, every downstream edit inherits the error. I agree that this is a real risk, especially for real-world PDFs and scans. However, the more load-bearing weakness in the reported evaluation is that the system is guided at inference time by SSIM-family feedback against the original chart, and the headline metric is SSIM against the original chart. Without a control showing that requested edits are actually applied, high SSIM can be achieved by a conservative system that barely changes anything. The paper provides no such control, no qualitative examples, no human study, and no statistical significance. This is not an accusation of dishonesty; it is a missing validation step. The abstract's specific '9–14%' improvement claim is also not supported by the numbers in Table 1, which strengthens the need for released code, prompts, and reproducibility checks. Because these issues are addressable with additional experiments and public artifacts, the appropriate verdict remains conditional rather than reject or accept. I do not see an internal inconsistency that would invalidate the framework outright; the concern is about whether the empirical evidence demonstrates the claimed editing capability.","tokens_in":4467,"tokens_out":4561,"duration_ms":44789,"concrete_test":"Run PlotEdit on a random 100-chart subset of ChartCraft with control requests: a no-op request ('keep the chart unchanged') and single-attribute edits with machine-checkable ground truth (e.g., 'change bar color to #FF0000', 'remove series C'). For each output, independently verify (i) SSIM against the original, (ii) actual pixel/table change, and (iii) whether the specific requested change is present, using color-histogram checks for style edits and DePlot-extracted tables for data edits. If no-op requests achieve SSIM nearly as high as real edits, or if high-SSIM real edits fail the presence checks, the current Table 1 comparison is not evidence of correct chart editing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on ChartCraft scores, but the architecture optimizes the same quantities used for evaluation. In Section 2, Visual Feedback iteratively adjusts Chart2Vision using MS-SSIM between original and replot, and Perceptual Fidelity Feedback revises edits using region-specific SSIM to the original chart. The headline metric SSIM is then computed against the original chart. A system that aggressively minimizes SSIM deviation from the input, without actually implementing the requested change, can therefore inflate SSIM. The paper reports no verification that requested edits are semantically realized, such as a data range actually filtered or a color actually changed, and it reports no error bars or statistical tests. Additionally, the abstract claims improvements of 9–14% over strong baselines, but Table 1 shows the largest visible gain over the strongest baseline (in-context learning) is about 8% in layout SSIM, with style, format, and data gains of roughly 1.5–4%. This weakens both the headline and the presumed mechanism that de-rendering accuracy plus multimodal feedback is the source of the gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PlotEdit, a five-LLM-agent pipeline for natural-language chart editing from PDF or scanned chart images: Chart2Table extracts the data table, Chart2Vision extracts visual attributes, Chart2Code retrieves rendering code, the Instruction Decomposition Agent splits user requests into executable steps, and the Multimodal Editing Agent applies data, style, and code edits. Three feedback mechanisms (code, visual, and numeric) refine the de-rendering agents, and a perceptual-fidelity feedback loop constrains the editing agent to preserve unchanged chart regions. The system is evaluated on the ChartCraft dataset against ChartLlama, ChartReformer, and in-context learning using SSIM, V AES, and RMS, with an ablation that removes multimodal feedback. The reported table shows PlotEdit achieving the best scores overall, with gains over the strongest baseline that are substantially smaller than the 9-14% claimed in the abstract.","tokens_in":4670,"tokens_out":4301,"duration_ms":41618,"significance":"If the quantitative claim were fully supported, PlotEdit would be a useful contribution to chart editing in PDFs and scanned documents, and its modular decomposition of chart de-rendering into table, style, and code components is a sensible design. The paper has concrete engineering strengths: the code feedback agent combines static AST checks with dynamic execution, the retrieval agents are given explicit modality-specific outputs, and the ablation (PlotEdit w/o MFA) provides some evidence about which components matter. However, the central claim of 9-14% improvement over strong baselines is not visible in Table 1, the metrics overlap with the optimization objectives used in the feedback loops, and the evaluation lacks the details needed to verify the comparison. These issues are load-bearing because the paper's main contribution is the reported performance.","major_comments":[{"comment":"The headline claim that PlotEdit outperforms strong baselines by 9-14% is not supported by Table 1. The largest per-category gain over in-context learning (the strongest baseline) is 8.0 points in layout SSIM (91.3 vs. 84.5); style, format, and data gains are roughly 1.5-4.4 points across metrics, and overall SSIM/VAES/RMS gains are 2.2/2.7/1.4 points. Please reconcile the abstract with the table, or report the calculation that yields 9-14%. This is load-bearing because the abstract's quantitative promise is the paper's main result.","section":"Abstract and §3, Table 1"},{"comment":"The evaluation is partially circular with the optimization objective. Visual Feedback iteratively adjusts Chart2Vision using MS-SSIM between the original and replot, and Perceptual Fidelity Feedback revises edits using region-specific SSIM against the original chart; SSIM against the original is then the headline evaluation metric. A system that minimizes SSIM deviation without implementing the requested change could therefore score well. The paper does not report any semantic verification that edits are actually realized (e.g., that a data range was filtered, a color changed, or a format converted according to the instruction). Please add an edit-success rate or human evaluation per edit type, or a control condition with a no-op instruction.","section":"§2, Visual Feedback and Perceptual Fidelity Feedback; §3, metrics"},{"comment":"The quantitative comparison is incomplete and not independently reproducible: ChartLlama's V AES and RMS cells are missing, no error bars or statistical tests are reported, the number and composition of ChartCraft test instances is not given, prompt templates and agent version numbers (e.g., GPT-4V/4o dates) are not specified, and ChartCraft itself is not in the reference list. These omissions block verification of the claim that PlotEdit significantly outperforms all baselines, especially for ChartLlama, where only SSIM is available.","section":"§3, Table 1"}],"minor_comments":[{"comment":"The phrase \"comprises of\" is nonstandard; \"comprises\" or \"consists of\" would be correct.","section":"§2"},{"comment":"The figure contains a great deal of textual detail that is illegible at the printed size; a higher-resolution version or a simplified schematic would help readers follow the agent flow.","section":"Figure 1"},{"comment":"The metric definitions are too terse: RMS is only named as \"Relative Mapping Similarity\" and V AES is only described as \"style precision,\" with no formulas or alignment details; this makes the table hard to interpret.","section":"§3"},{"comment":"The phrase \"significantly outperforms\" is used without any statistical test or error bar; please qualify it or supply uncertainty estimates.","section":"§3"},{"comment":"The feedback loops are described as running \"until satisfactory results or exhaustion of max trials,\" but the maximum number of trials and the stopping criterion are never specified.","section":"§2"},{"comment":"The application claim of \"enhancing accessibility for visually challenged users\" is not evaluated with any user study or accessibility-specific metric; it is presented as a consequence of color/contrast editing but no evidence is given.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually short for a full research paper, and the central quantitative claim is inconsistent with its own table. The architecture is interesting and the ablation is a positive sign, but the evaluation needs substantial expansion (dataset details, prompt/version details, error bars, semantic verification of edits, and a corrected headline) before the paper can be assessed for publication. I would not reject on mechanism alone, but the current evidence does not support the abstract's 9-14% claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PlotEdit is a competent engineering contribution to a narrow but real problem: editing charts that only exist as images in PDFs. The new part is the orchestration—five LLM agents that de-render a chart into table/style/code, decompose the user request, edit, and then iteratively correct themselves using code, numeric, and visual feedback. The ablation (PlotEdit w/o MFA) suggests the feedback loop is doing useful work. That is a legitimate contribution to the chart-editing subfield, and the accessibility motivation is genuine. The paper also targets the right failure mode: LLM prompting fails at de-rendering, so self-correction is a sensible instinct. The references are the right ones—ChartReformer, ChartLlama, DePlot—and I don't see a citation problem.\n\nBut the evidence as printed does not support the headline. The evaluation is a single table with no error bars, no statistical tests, no dataset splits, no prompt or code release, and ChartLlama's V AES/RMS values are blank. The abstract's 9–14% improvement over strong baselines is not in the table: the largest visible relative gain over in-context learning is about 8% (layout SSIM), with style/format/data gains mostly in the 1.5–4% range. That is a real overstatement.\n\nThe deeper problem is metric design. The Visual Feedback and Perceptual Fidelity Feedback agents use MS-SSIM/SSIM to keep the edited chart close to the original, and the headline evaluation metric is SSIM against the original. A system that aggressively preserves the input can therefore inflate SSIM without actually implementing the requested edit. The paper reports no semantic verification—no check that a data range was filtered or a color changed. This does not sink the paper, because the comparison against external baselines on the same metric is still informative, but it means the gains are not yet a measure of edit correctness. The whole pipeline also inherits GPT-4V de-rendering errors; the feedback loops mitigate this, but the paper doesn't show how often de-rendering fails.\n\nWho should read this: people working on chart understanding/editing, and anyone designing evaluation for LLM-agent pipelines. It is a good case study in why pixel-similarity metrics are not enough. I would not cite it in my own work until the eval is tightened. For peer review: yes, it deserves referee time, but only with a request for code/data, confidence intervals, and a semantic edit-correctness check—and a corrected abstract.","headline":"Sensible multi-agent chart-editing system, but the evaluation is too thin and partly metric-gamed to support the paper's headline numbers.","tokens_in":5182,"tokens_out":4647,"would_cite":false,"duration_ms":40898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chart images in PDFs can be edited by natural-language instructions, without the source data or code, by decomposing the chart, editing its parts, and re-rendering it through coordinated multimodal agents.","keywords":["chart editing","LLM agents","chart de-rendering","multimodal feedback","self-reflection","PDF accessibility","natural language editing","visual fidelity"],"falsifier":"Take a set of real scanned or low-resolution chart images whose original data and rendering code are known, run PlotEdit's de-rendering agents on them, and compare the extracted table and code against the ground truth; if the extracted representation cannot reproduce the original chart within the allowed feedback rounds, then the edit quality predicted by the framework collapses on the precise input type the paper targets.","tokens_in":4293,"feed_emoji":"📊","tokens_out":8520,"duration_ms":82006,"temperature":0.7,"pith_summary":"PlotEdit claims that charts stored as images inside PDFs or scans can be edited by natural-language instructions even when the original data table, style definitions, and source code are gone. The paper builds a five-agent pipeline that first de-renders the chart into a data table, visual attributes, and Python plotting code, then decomposes the user's request into individual edits, applies those edits to the appropriate component, and re-renders the chart. Three self-reflection loops, based on code validation, image-region comparison, and data-summary comparison, catch de-rendering errors before editing. On a benchmark of chart edits, the pipeline is reported to outperform direct in-context prompting and prior chart-editing models across style, layout, format, and data edits. The motivation is practical: making chart editing accessible to visually impaired readers and to non-experts who do not have the source files or specialized software.","feed_headline":"Five AI agents edit PDF charts from plain language","feed_subtitle":"A self-reflecting pipeline turns any chart image into data, style, and code, edits it, then re-renders it faithfully.","key_machinery":"The load-bearing mechanism is a de-render, edit, re-render loop. Three retrieval agents build the editable representation: Chart2Table extracts the data table through chain-of-thought prompting, Chart2Vision records colors, fonts, markers, and legend mappings in JSON, and Chart2Code generates Python visualization code. Three additional feedback agents then refine that representation before any edit is made: code feedback parses and executes the code to catch errors, visual feedback compares original and replotted images in local regions using multi-scale structural similarity, and numeric feedback compares chart summaries and plot-to-table statistics to catch data mismatches. The Instruction Decomposition Agent turns the user request into a step sequence, the Multimodal Editing Agent applies those steps to the table, style JSON, or code, and a final perceptual-fidelity check confirms that unchanged regions stay untouched.","core_discovery":"Put in the paper's own terms, the central claim is that the bottleneck in language-driven chart editing is not the model's ability to apply an edit but its ability to de-render the chart image faithfully beforehand. PlotEdit therefore converts the chart into three editable artifacts, a data table, a JSON of visual attributes, and executable plotting code, and only then lets an editing agent modify them. The claimed payoff is that this decomposition, guided by multimodal feedback, produces edited charts that preserve the original's visual integrity while carrying out the requested change, and does so more reliably than asking a single model to reason directly from pixels. The paper further claims that the feedback loop is what separates the approach from weaker baselines, since it turns de-rendering mistakes into correctable errors instead of silent hallucinations.","pith_inferences":["Editorial inference: if the loop transfers to other image-to-code tasks, the same de-render, verify, edit, re-render scaffold could apply to diagrams, maps, or screenshots, since the three feedback signals are not chart-specific.","Editorial inference: the reported advantage is tied to the current de-rendering capability of the underlying multimodal models; the durable contribution may be the self-reflection architecture, whose value should grow as the base models improve.","Editorial inference: a direct test of the accessibility claim would measure whether screen-reader output or novice completion time actually improves; the paper's evidence is image-similarity and data-fidelity metrics rather than these downstream outcomes."],"forward_implications":["Charts in PDFs and scans become editable without recovering the original spreadsheet or script, removing the main practical barrier to editing legacy documents.","De-rendering errors that would otherwise corrupt an edited chart become visible and correctable, because the same image is re-rendered and compared with the input.","The data table produced during de-rendering can serve as an accessibility artifact, such as textual or machine-readable chart content.","A user who can describe a change in plain language can apply style, layout, format, and data edits without learning plotting libraries or design tools."],"supporting_citations":[{"why":"Supplies the prior chart-editing model and evaluation protocol that PlotEdit must beat, especially on data-centric and layout edits.","marker":"[10]"},{"why":"Provides the multimodal chart generation baseline whose de-rendering failures motivate the need for an agentic feedback loop.","marker":"[1]"},{"why":"Supplies the plot-to-table translation used for numeric consistency feedback and for the data-fidelity evaluation metric.","marker":"[2]"},{"why":"Defines the structural similarity metric used for visual feedback and final evaluation.","marker":"[7]"},{"why":"Defines multi-scale structural similarity used to localize visual mismatches in regions of the chart.","marker":"[8]"},{"why":"Supplies the chain-of-thought prompting technique used by decomposition and extraction agents to turn requests into executable steps.","marker":"[9]"}],"fun_headline_variants":["Edit PDF charts by describing the change in words","Five AI agents de-render chart images for faithful edits","From pixel to code to edit: AI chart pipeline","Tell AI to change a chart, it rebuilds from extracted code","Chart editing via language: extract data, style, code first"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a vision-language model can reliably decompose a chart image into a correct data table, style attributes, and code; if that de-rendering step is inaccurate, every edit made downstream inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["Edit PDF charts by describing the change in words","Five AI agents de-render chart images for faithful edits","From pixel to code to edit: AI chart pipeline","Tell AI to change a chart, it rebuilds from extracted code","Chart editing via language: extract data, style, code first"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1339,"prompt_tokens":871,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":487,"tokens_out":468,"duration_ms":4756,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:29:26.865534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of real scanned or low-resolution chart images whose original data and rendering code are known, run PlotEdit's de-rendering agents on them, and compare the extracted table and code against the ground truth; if the extracted representation cannot reproduce the original chart within the allowed feedback rounds, then the edit quality predicted by the framework collapses on the precise input type the paper targets.","supporting_citations":[{"cited_title":"In: International Conference on Document Analysis and Recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the prior chart-editing model and evaluation protocol that PlotEdit must beat, especially on data-centric and layout edits."},{"cited_title":"IEEE transactions on image processing 13(4), 600--612 (2004)","cited_arxiv_id":null,"evidence_quote":"Defines the structural similarity metric used for visual feedback and final evaluation."},{"cited_title":"In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003","cited_arxiv_id":null,"evidence_quote":"Defines multi-scale structural similarity used to localize visual mismatches in regions of the chart."},{"cited_title":"Advances in neural information processing systems 35, 24824--24837 (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting technique used by decomposition and extraction agents to turn requests into executable steps."}],"review_version":1}