{"id":"c4e3ab5c-7b27-4d7a-97a4-8e194a4ccdb9","arxiv_id":"2412.02764","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PandasPlotBench is a 175-task benchmark showing that LLMs generate good Matplotlib and Seaborn plots from brief instructions but struggle with Plotly.","lead":"This paper introduces PandasPlotBench, a curated set of 175 tasks that tests how well large language models turn a data table and a short instruction into working plotting code. It finds that shortening user instructions barely hurts output quality, while models still make many mistakes with the Plotly library.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Public Matplotlib gallery as source leaves the 'leakage-resistant' claim vulnerable: high Matplotlib and task-shortening scores may reflect memorized gallery code, not capability.","rationale":"The reader's weakest assumption, data leakage from the public Matplotlib gallery, is also the most load-bearing concern. The benchmark's advertised property and the task-compression conclusion both depend on it: if models retrieve memorized gallery code, the high Matplotlib scores and the 'minimal effect' of shortening tasks overestimate real plotting capability. Section V's explicit admission strengthens this concern, and the conclusion's unqualified 'free from data leakage' is an internal tension. I agree with the reader's conditional verdict: the concern is real and fixable but does not invalidate the benchmark outright, especially because the Plotly finding and the executable-code infrastructure are independently useful. The proposed twin-dataset test would settle the question with the released code. Since the reader already assigned CONDITIONAL, no verdict change is needed.","tokens_in":8074,"tokens_out":6501,"duration_ms":72690,"concrete_test":"Build a decontamination twin of each of the 175 tasks: keep the plot's visual semantics but anonymize surface features by renaming columns to meaningless labels, applying an affine transform or normalization to numeric data (or reordering rows), and rewriting the task text to match the transformed schema. Run GPT-4o under the identical benchmark protocol on originals and twins. If the mean task-based score on twins falls more than about 5 points below originals, or a paired bootstrap 95% confidence interval excludes a negligible difference, the original scores are inflated by memorization of public gallery examples; if the interval includes zero or a small effect, the leakage concern is refuted. Report the per-run standard deviation across the five runs for both sets to also address the missing variance in Table III.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central premise that PandasPlotBench is leakage-resistant (Conclusion, Section VI) is undercut by the dataset's construction: Section II-A sources all data from the public Matplotlib gallery, and the ground-truth plotting code is a manually corrected split of those public scripts. Public gallery code and its hard-coded datasets are exactly the kind of material found in LLM training corpora. The authors concede in Section V that 'models may still exhibit bias towards publicly available ground-truth Matplotlib code,' which is in tension with the unqualified 'free from data leakage' claim in Section VI. Synthetic task prompts and CSV files are a surface transformation; they do not prevent a model from recognizing the first five rows or column names of a memorized gallery example and retrieving the associated plot. This matters because the headline 'minimal effect of task shortening' (Table III: task score 89 to 85) is measured only with GPT-4o on Matplotlib. If the model can complete the plot from memorized gallery data, the single-sentence condition is not actually testing instruction following; it is testing retrieval. The 'No task' control (score 36) shows the task is needed, but it does not show whether full and short tasks differ because of genuine language understanding. The Plotly finding (22% incorrect code) is less exposed, since Plotly code is not in the gallery source, but the benchmark's main validity claim and the UI recommendation rest on the Matplotlib results. No decontamination analysis (overlap, membership probing, or held-out variants) is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PandasPlotBench, a human-curated benchmark of 175 tasks for evaluating LLMs' ability to generate plotting code from a Pandas DataFrame given natural-language instructions. The data are derived from the Matplotlib gallery: scripts are split into data-preparation and plotting parts, manually verified, and accompanied by detailed, short, and single-sentence task prompts. The authors evaluate several proprietary and open LLMs, report per-library performance for Matplotlib, Seaborn, and Plotly, and study the effect of task compression. The central findings are that task shortening has minimal effect on plot quality (task-based score 89 vs. 85 in Table III), and that models struggle noticeably with Plotly (22% incorrect code in Table II). The paper also describes two scoring methods, visual and task-based, both performed by a GPT-4o judge, with a single-author human correlation check.","tokens_in":8324,"tokens_out":3324,"duration_ms":35219,"significance":"If the methodological concerns can be resolved, PandasPlotBench is a useful complement to existing visualization benchmarks such as MatPlotBench and Plot2Code: it provides more data points with actual CSV files, spans three plotting libraries, and includes multiple task-length variants. The authors have released the dataset and code, which supports reproducibility and further use. The finding that concise instructions suffice for high-quality plots, if substantiated, would inform UI design for LLM-based visualization assistants. The paper is transparent about several limitations in Section V, which is commendable. However, the benchmark's validity hinges on two load-bearing issues: the leakage-resistance claim and the reliability of the LLM judge, both of which currently lack sufficient evidence.","major_comments":[{"comment":"The claim that the benchmark is \"free from data leakage\" (Section VI) is in direct tension with the dataset construction described in Section II-A and the admission in Section V that \"models may still exhibit bias towards publicly available ground-truth Matplotlib code.\" Since the ground-truth code and the underlying DataFrames are derived from the public Matplotlib gallery, and the gallery scripts and their hard-coded datasets are very likely present in LLM training corpora, the synthetic task prompts and CSV files are only a surface transformation. The \"No task\" control in Table III shows that the task matters, but it does not distinguish genuine instruction-following from retrieval of a memorized gallery example. To support the leakage-resistance claim, the authors should either remove or substantially soften the conclusion, or provide a concrete contamination test (e.g., comparing model performance on original versus column-renamed or row-permuted DataFrames, or measuring overlap with training data). This concern is load-bearing because the headline finding about task shortening is measured on Matplotlib, the library most exposed to this issue.","section":"Section II-A, Section V, Section VI"},{"comment":"The evaluation uses GPT-4o as both the best-performing generation model (Table I) and the judge for all scoring, while the tasks themselves were generated with GPT-4V. The only human validation is a single-author sanity check on one setup (Pearson r = 0.85 for task-based, 0.66 for visual scores, Section II-C). This creates a real risk of self-preference bias in the central claim that task shortening has minimal effect. The paper asserts in Section V that the bias is \"negligible,\" but provides no evidence for that assertion. I recommend adding a multi-annotator human study with inter-rater reliability statistics, or at minimum a cross-validation of the judge against one or more independent LLM judges, and reporting agreement between judges. Without such evidence, the task-length comparison in Table III is not robustly established.","section":"Section II-C and Section III-C"},{"comment":"The experiments are run five times per factor (Section III), yet the tables report only means and percentages, with no confidence intervals, standard deviations, or significance tests. For example, the difference between the basic task (task score 89) and the single-sentence task (task score 85) in Table III could plausibly be within run-to-run variance. Similarly, the \"good (≥75)\" ratios are point estimates. To support the claim that task shortening has \"minimal effect,\" the authors should report per-run variation and perform a statistical comparison across the five runs. This is a load-bearing issue for the practical UI recommendation drawn from this result.","section":"Tables I-III"}],"minor_comments":[{"comment":"There is a typographical error: \"In contrast, en the task was omitted\" should read \"In contrast, when the task was omitted.\"","section":"Section III-C"},{"comment":"The affiliation line contains a typo: \"Y aroslav Golubev\" and \"T echnology\" should be \"Yaroslav Golubev\" and \"Technology.\"","section":"Author affiliations"},{"comment":"The sentence \"This quick check allows us to trust in the meaningful results of the benchmark\" overstates the evidentiary weight of a single-author correlation; consider rephrasing to reflect that this is a sanity check that requires further validation.","section":"Section II-C"},{"comment":"The \"Wrong library\" column shows 1 for Seaborn but no explanation is given for what happened in that case; a brief note would improve clarity.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful benchmark with a clear gap in existing evaluation tools, and the authors have been transparent about many limitations. The main risk is that the headline claims—especially leak-resistance and the minimal effect of task shortening—are currently supported only by the authors' own assumptions and a self-referential LLM-judge setup. These issues are fixable within the scope of the manuscript by adding contamination checks, multi-annotator validation, and variance reporting. I therefore see major revision as appropriate rather than rejection. I would also gently advise the authors to align the strength of the conclusion with the evidence in Section V, since the current inconsistency may draw scrutiny from reviewers and readers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper if you care about LLM benchmarks for data visualization. PandasPlotBench is a real contribution: 175 human-curated tasks, each with an actual CSV file, a natural-language task, and ground-truth Matplotlib code. That alone puts it ahead of MatPlotBench (25 data files) and Plot2Code (no data files). The authors also ship the dataset and evaluation code, which makes the whole thing independently checkable. Good.\n\nThe experimental findings are reasonable in direction: GPT-4o and Claude 3.5 Sonnet top the charts, smaller Llama models fall apart, Plotly is a clear weak spot (22% incorrect code), and task compression from a detailed prompt to a single sentence barely moves task-based scores (89 to 85). The 'No task' control is a nice sanity check: scores collapse to 36, so the task text is doing real work.\n\nNow the soft spots, in proportion. The biggest is the leakage claim. The dataset is built from the public Matplotlib gallery. The authors have modified the scripts and synthetically generated the prompts and CSVs, but that is not the same as being free from data leakage. In Section V they admit models 'may still exhibit bias towards publicly available ground-truth Matplotlib code,' yet the conclusion says the benchmark is 'free from data leakage by virtue of being synthetically generated.' That is a real tension. If a model has memorized a gallery script, it can retrieve a plausible plot from the column names or first five rows of the CSV, which would inflate Matplotlib scores and make the task-compression result look better than it is. No decontamination analysis is reported. This is fixable—run a membership probe, test with held-out variants, or at least soften the claim—but as written, the central validity claim is overreaching.\n\nThe second concern is the evaluation stack. The judge is GPT-4o, and GPT-4o is also the best-performing model. That is circular. The human correlation check (Pearson r = 0.85 for task-based scores) is a good mitigation, but it was done by the first author alone on one setup. Also, the five runs are averaged with no standard deviations or error bars; the 'minimal effect' of task compression should be backed by a bit of statistics, even simple confidence intervals.\n\nWho is this for? People building LLM-based visualization assistants, and anyone designing code-generation benchmarks. The paper is a solid engineering contribution with a clear evaluation protocol and open resources. It deserves a serious referee. I would recommend conditional acceptance: the benchmark is worth publishing, but the leakage claim needs to be tempered and the statistical reporting tightened. The fixes are straightforward and the paper will be better for them.","headline":"PandasPlotBench is a useful, reproducible benchmark for LLM plotting-code generation, but its 'leakage-resistant' claim is overstated and the evaluation needs a bit more statistical backbone.","tokens_in":8909,"tokens_out":1851,"would_cite":true,"duration_ms":20110,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 175-task benchmark shows LLMs keep most of their plotting skill when instructions drop to one sentence, and exposes Plotly as their weak spot.","keywords":["LLM code generation","data visualization","Pandas","benchmark","Matplotlib","Seaborn","Plotly","task compression"],"falsifier":"Build a matched set of plotting tasks whose data and plotting intentions are private and were created after the tested models' training, then score the same models on both sets; if task-based scores fall well below the 85-89 range on the private set, the leakage-resistance assumption is the cause and the reported capability is inflated.","tokens_in":7868,"feed_emoji":"📊","tokens_out":9002,"duration_ms":83438,"temperature":0.7,"pith_summary":"PandasPlotBench is a human-curated set of 175 tasks that test whether a large language model can turn a Pandas DataFrame and a natural-language instruction into working plotting code. The paper argues the benchmark is leakage-resistant because the tasks and data files are synthetically generated rather than copied from model training corpora, and uses it to measure how task compression and plotting-library choice affect output. The central finding is that cutting a detailed instruction down to a single sentence barely lowers task-based scores (85 versus 89 on a 0-100 scale) when the model receives a structured description of the DataFrame. A second finding is that Matplotlib and Seaborn are handled well, while 22% of Plotly tasks produce incorrect code, mostly from misuse of the Plotly API. If these results hold, a data-visualization assistant can accept very short user requests without much quality loss, and Plotly support is a targeted place to improve models.","feed_headline":"LLMs lose little skill when plot prompts shrink to one sentence","feed_subtitle":"New 175-task benchmark confirms short instructions suffice and exposes Plotly as models' weak spot.","key_machinery":"The central object is PandasPlotBench itself, a dataset of 175 data points; each point consists of a CSV DataFrame, a small data-loading script, a ground-truth Matplotlib plot and code, and three versions of the plotting task (detailed, shortened to 2-3 sentences, and single-sentence). The evaluation pipeline runs the generated code in a Jupyter cell, extracts the plot image, and has a multimodal judge model score it on a 0-100 scale for visual similarity to the ground truth and for adherence to the task text. The benchmark's design separates each prompt into framework setup, task, and styling options, which is what lets the authors vary library choice, task compression, and DataFrame-description format while holding the underlying data fixed.","core_discovery":"The paper's claim is that PandasPlotBench is a leakage-resistant evaluation of LLM plotting-code generation, and that its first results establish two regularities: shortening task descriptions has minimal effect on plot quality when DataFrame descriptions are informative, and the least popular of the three tested libraries, Plotly, exposes the largest model gap. The evidence is 175 tasks built from public Matplotlib gallery scripts, a Jupyter-based execution pipeline, and a multimodal judge that rates plots on task adherence and visual similarity, with a single-expert human scoring run validating the task-based scores at a 0.85 Pearson correlation. The headline numbers are a task-based mean score of 89 for the best model on the full tasks versus 85 on single-sentence tasks, and a 22% incorrect-code rate for Plotly versus 1.8% for Matplotlib.","pith_inferences":["Because the tasks derive from the public Matplotlib gallery, a strict test of leakage resistance would be to score the same models on a matched set of plotting tasks built from private or newly authored plots; a large score gap would indicate that part of the reported capability is memorization rather than transferable skill.","The minimal penalty for short tasks suggests that automatic DataFrame summarization, such as a compact preview block, could replace user effort in prompt construction, a direction the paper's experiments support but do not directly optimize.","The 22% Plotly failure rate, if it tracks the library's representation in training corpora, predicts that models will improve unevenly across libraries over time; rerunning the same 175 tasks as new model versions appear would quantify that drift.","Since the human sanity check involved a single scorer, a multi-annotator study on the same dataset would turn the 0.85 correlation into a validated, stable estimate of judge reliability."],"forward_implications":["If the task-shortening result generalizes, user interfaces for data visualization can accept one-sentence requests as long as they supply a structured data summary, without expecting a large quality penalty.","Plotly is the current bottleneck: even the strongest model fails to produce executable Plotly code on 22% of tasks, so training data and API-focused evaluation for less common libraries should be a priority.","Open-weight models at 70B-405B scale come close to the best proprietary scores, while 1B-8B models frequently produce broken code, suggesting model scale remains decisive for plotting-code generation.","Task-based judging with an LLM judge aligns with human judgment strongly enough (0.85 correlation) to serve as a scalable scoring method for future plotting benchmarks."],"supporting_citations":[{"why":"Supplies the evaluation methodology and judging prompts that PandasPlotBench adapts, including task-based and visual scoring on a 0-100 scale.","marker":"[4]"},{"why":"Source of the 501 Matplotlib plotting scripts from which the 175 benchmark tasks and DataFrames are derived after filtering, splitting, and manual verification.","marker":"[21]"},{"why":"The model used to split each gallery script into a data-generating part and a plotting part during dataset construction.","marker":"[22]"},{"why":"The vision-language model used to generate the detailed task instructions from the plotting code, DataFrame metadata, and plot image.","marker":"[23]"},{"why":"The code-similarity scorer whose scores showed no correlation with visual or task scores and was therefore rejected in favor of judge-based scoring.","marker":"[24]"},{"why":"An earlier data-science code benchmark whose short Matplotlib solutions motivate the need for more realistic, full-task plotting evaluation.","marker":"[6]"},{"why":"A large-scale visualization benchmark targeting SQL and Vega-Lite-like languages, marking the gap in text-to-Python-plotting evaluation this work fills.","marker":"[9]"}],"fun_headline_variants":["Short prompts barely hurt LLM plotting, Plotly does","PandasPlotBench: LLMs ace Matplotlib, struggle with Plotly","LLM plot code: concise instructions suffice, Plotly exposes gaps","New benchmark: LLMs plot well with short commands, except Plotly","Plotly stumps LLMs even as short prompts keep plot quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's scores measure general plotting skill only if models have not memorized the public Matplotlib gallery scripts the tasks were derived from, and the paper concedes that models may still show bias toward that publicly available ground-truth code.","fun_headline_variants_meta":{"raw":{"variants":["Short prompts barely hurt LLM plotting, Plotly does","PandasPlotBench: LLMs ace Matplotlib, struggle with Plotly","LLM plot code: concise instructions suffice, Plotly exposes gaps","New benchmark: LLMs plot well with short commands, except Plotly","Plotly stumps LLMs even as short prompts keep plot quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000552,"raw_usage":{"total_tokens":2613,"prompt_tokens":907,"completion_tokens":1706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1614}},"tokens_in":523,"tokens_out":1706,"duration_ms":13979,"temperature":1.0,"reasoning_tokens":1614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:06:40.375721+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a matched set of plotting tasks whose data and plotting intentions are private and were created after the tested models' training, then score the same models on both sets; if task-based scores fall well below the 85-89 range on the private set, the leakage-resistance assumption is the cause and the reported capability is inflated.","supporting_citations":[{"cited_title":"MatPlotLib Gallery,","cited_arxiv_id":null,"evidence_quote":"Source of the 501 Matplotlib plotting scripts from which the 175 benchmark tasks and DataFrames are derived after filtering, splitting, and manual verification."},{"cited_title":"GPT-4V(ision) System Card,","cited_arxiv_id":null,"evidence_quote":"The vision-language model used to generate the detailed task instructions from the plotting code, DataFrame metadata, and plot image."},{"cited_title":"DS-1000: A natural and reliable benchmark for data science code generation,","cited_arxiv_id":null,"evidence_quote":"An earlier data-science code benchmark whose short Matplotlib solutions motivate the need for more realistic, full-task plotting evaluation."}],"review_version":1}