Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 175-task benchmark shows LLMs keep most of their plotting skill when instructions drop to one sentence, and exposes Plotly as their weak spot.

desk verdict PandasPlotBench is a useful, reproducible benchmark for LLM plotting-code generation, but its 'leakage-resistant' claim is overstated and the evaluation needs a bit more statistical backbone. read the letter →

arxiv 2412.02764 v2 pith:DMTQKLKY submitted 2024-12-03 cs.SE cs.AIcs.LG

Timur Galimzyanov , Sergey Titov , Yaroslav Golubev , Egor Bogomolov This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords LLMcodegenerationdatavisualizationPandasbenchmarkMatplotlibSeabornPlotlytaskcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PandasPlotBench is a human-curated set of 175 tasks that test whether a large language model can turn a Pandas DataFrame and a natural-language instruction into working plotting code. The paper argues the benchmark is leakage-resistant because the tasks and data files are synthetically generated rather than copied from model training corpora, and uses it to measure how task compression and plotting-library choice affect output. The central finding is that cutting a detailed instruction down to a single sentence barely lowers task-based scores (85 versus 89 on a 0-100 scale) when the model receives a structured description of the DataFrame. A second finding is that Matplotlib and Seaborn are handled well, while 22% of Plotly tasks produce incorrect code, mostly from misuse of the Plotly API. If these results hold, a data-visualization assistant can accept very short user requests without much quality loss, and Plotly support is a targeted place to improve models.

What carries the argument

The central object is PandasPlotBench itself, a dataset of 175 data points; each point consists of a CSV DataFrame, a small data-loading script, a ground-truth Matplotlib plot and code, and three versions of the plotting task (detailed, shortened to 2-3 sentences, and single-sentence). The evaluation pipeline runs the generated code in a Jupyter cell, extracts the plot image, and has a multimodal judge model score it on a 0-100 scale for visual similarity to the ground truth and for adherence to the task text. The benchmark's design separates each prompt into framework setup, task, and styling options, which is what lets the authors vary library choice, task compression, and DataFrame-description format while holding the underlying data fixed.

What would settle it

Build a matched set of plotting tasks whose data and plotting intentions are private and were created after the tested models' training, then score the same models on both sets; if task-based scores fall well below the 85-89 range on the private set, the leakage-resistance assumption is the cause and the reported capability is inflated.

Watch

Extended reading notes

Core claim

The paper's claim is that PandasPlotBench is a leakage-resistant evaluation of LLM plotting-code generation, and that its first results establish two regularities: shortening task descriptions has minimal effect on plot quality when DataFrame descriptions are informative, and the least popular of the three tested libraries, Plotly, exposes the largest model gap. The evidence is 175 tasks built from public Matplotlib gallery scripts, a Jupyter-based execution pipeline, and a multimodal judge that rates plots on task adherence and visual similarity, with a single-expert human scoring run validating the task-based scores at a 0.85 Pearson correlation. The headline numbers are a task-based mean score of 89 for the best model on the full tasks versus 85 on single-sentence tasks, and a 22% incorrect-code rate for Plotly versus 1.8% for Matplotlib.

Load-bearing premise

The benchmark's scores measure general plotting skill only if models have not memorized the public Matplotlib gallery scripts the tasks were derived from, and the paper concedes that models may still show bias toward that publicly available ground-truth code.

Editorial extensions

If this is right

  • If the task-shortening result generalizes, user interfaces for data visualization can accept one-sentence requests as long as they supply a structured data summary, without expecting a large quality penalty.
  • Plotly is the current bottleneck: even the strongest model fails to produce executable Plotly code on 22% of tasks, so training data and API-focused evaluation for less common libraries should be a priority.
  • Open-weight models at 70B-405B scale come close to the best proprietary scores, while 1B-8B models frequently produce broken code, suggesting model scale remains decisive for plotting-code generation.
  • Task-based judging with an LLM judge aligns with human judgment strongly enough (0.85 correlation) to serve as a scalable scoring method for future plotting benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the tasks derive from the public Matplotlib gallery, a strict test of leakage resistance would be to score the same models on a matched set of plotting tasks built from private or newly authored plots; a large score gap would indicate that part of the reported capability is memorization rather than transferable skill.
  • The minimal penalty for short tasks suggests that automatic DataFrame summarization, such as a compact preview block, could replace user effort in prompt construction, a direction the paper's experiments support but do not directly optimize.
  • The 22% Plotly failure rate, if it tracks the library's representation in training corpora, predicts that models will improve unevenly across libraries over time; rerunning the same 175 tasks as new model versions appear would quantify that drift.
  • Since the human sanity check involved a single scorer, a multi-annotator study on the same dataset would turn the 0.85 correlation into a validated, stable estimate of judge reliability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces PandasPlotBench, a human-curated benchmark of 175 tasks for evaluating LLMs' ability to generate plotting code from a Pandas DataFrame given natural-language instructions. The data are derived from the Matplotlib gallery: scripts are split into data-preparation and plotting parts, manually verified, and accompanied by detailed, short, and single-sentence task prompts. The authors evaluate several proprietary and open LLMs, report per-library performance for Matplotlib, Seaborn, and Plotly, and study the effect of task compression. The central findings are that task shortening has minimal effect on plot quality (task-based score 89 vs. 85 in Table III), and that models struggle noticeably with Plotly (22% incorrect code in Table II). The paper also describes two scoring methods, visual and task-based, both performed by a GPT-4o judge, with a single-author human correlation check.

Significance. If the methodological concerns can be resolved, PandasPlotBench is a useful complement to existing visualization benchmarks such as MatPlotBench and Plot2Code: it provides more data points with actual CSV files, spans three plotting libraries, and includes multiple task-length variants. The authors have released the dataset and code, which supports reproducibility and further use. The finding that concise instructions suffice for high-quality plots, if substantiated, would inform UI design for LLM-based visualization assistants. The paper is transparent about several limitations in Section V, which is commendable. However, the benchmark's validity hinges on two load-bearing issues: the leakage-resistance claim and the reliability of the LLM judge, both of which currently lack sufficient evidence.

major comments (3)
  1. [Section II-A, Section V, Section VI] The claim that the benchmark is "free from data leakage" (Section VI) is in direct tension with the dataset construction described in Section II-A and the admission in Section V that "models may still exhibit bias towards publicly available ground-truth Matplotlib code." Since the ground-truth code and the underlying DataFrames are derived from the public Matplotlib gallery, and the gallery scripts and their hard-coded datasets are very likely present in LLM training corpora, the synthetic task prompts and CSV files are only a surface transformation. The "No task" control in Table III shows that the task matters, but it does not distinguish genuine instruction-following from retrieval of a memorized gallery example. To support the leakage-resistance claim, the authors should either remove or substantially soften the conclusion, or provide a concrete contamination test (e.g., comparing model performance on original versus column-renamed or row-permuted DataFrames, or measuring overlap with training data). This concern is load-bearing because the headline finding about task shortening is measured on Matplotlib, the library most exposed to this issue.
  2. [Section II-C and Section III-C] The evaluation uses GPT-4o as both the best-performing generation model (Table I) and the judge for all scoring, while the tasks themselves were generated with GPT-4V. The only human validation is a single-author sanity check on one setup (Pearson r = 0.85 for task-based, 0.66 for visual scores, Section II-C). This creates a real risk of self-preference bias in the central claim that task shortening has minimal effect. The paper asserts in Section V that the bias is "negligible," but provides no evidence for that assertion. I recommend adding a multi-annotator human study with inter-rater reliability statistics, or at minimum a cross-validation of the judge against one or more independent LLM judges, and reporting agreement between judges. Without such evidence, the task-length comparison in Table III is not robustly established.
  3. [Tables I-III] The experiments are run five times per factor (Section III), yet the tables report only means and percentages, with no confidence intervals, standard deviations, or significance tests. For example, the difference between the basic task (task score 89) and the single-sentence task (task score 85) in Table III could plausibly be within run-to-run variance. Similarly, the "good (≥75)" ratios are point estimates. To support the claim that task shortening has "minimal effect," the authors should report per-run variation and perform a statistical comparison across the five runs. This is a load-bearing issue for the practical UI recommendation drawn from this result.
minor comments (4)
  1. [Section III-C] There is a typographical error: "In contrast, en the task was omitted" should read "In contrast, when the task was omitted."
  2. [Author affiliations] The affiliation line contains a typo: "Y aroslav Golubev" and "T echnology" should be "Yaroslav Golubev" and "Technology."
  3. [Section II-C] The sentence "This quick check allows us to trust in the meaningful results of the benchmark" overstates the evidentiary weight of a single-author correlation; consider rephrasing to reflect that this is a sanity check that requires further validation.
  4. [Table II] The "Wrong library" column shows 1 for Seaborn but no explanation is given for what happened in that case; a brief note would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark results are empirical measurements, and the LLM-judge overlap is a bias risk, not a by-construction reduction.

full rationale

The paper's central claims (task compression has minimal effect; Plotly underperforms) are empirical observations from running models on a benchmark, not derived quantities that reproduce their inputs. The dataset is constructed from the public Matplotlib gallery and converted into DataFrames and task prompts (Section II-A), but the reported scores come from executing generated code and having a separate judge prompt score adherence to the task; the ground-truth code and task prompts are not fitted to the reported scores. The use of GPT-4o as both the best-performing generator and the judge (Section II-C, Table I) is a potential measurement bias, and the paper itself flags possible OpenAI bias in Section V; however, this does not make any score equivalent to its input by construction. Likewise, the 'free from data leakage' claim in the Conclusion is in tension with the gallery-sourced data and the Section V caveat, but that is a validity and correctness concern, not a circularity. No fitted parameter is renamed as a prediction, no load-bearing uniqueness theorem is imported from self-citations, and the self-citations [18]-[20] are resource links rather than argumentative premises. Therefore, no circular step can be exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no new theoretical entities. Its central claims rest on domain assumptions about LLM judge reliability, data leakage, and code-execution-based evaluation. The most fragile are the leakage-free claim and the single-judge scoring, both partially acknowledged by the authors.

assumptions (4)
  • domain assumption GPT-4o judge scores approximate human judgment of plot-task adherence.
    The evaluation uses GPT-4o as the judge for all scores. The paper validates this with a single human scorer on one setup (Section II-C), showing r=0.85 for task-based scores, but this is not a broad, multi-expert validation.
  • domain assumption The synthetic tasks and data files have not been seen by LLMs during training.
    Section V states the benchmark is 'free from data leakage by virtue of being synthetically generated,' but the data is derived from the public Matplotlib gallery, which is likely in training data. The authors acknowledge this risk.
  • domain assumption Executing code in Jupyter and ignoring errors when a plot is produced is a valid measure of code correctness.
    Section II-C defines incorrect code as cells that did not result in a plot, and errors are ignored when plots are also returned. This treats partial or error-prone code as acceptable, which may overestimate correctness.
  • domain assumption The Matplotlib gallery is a representative source of typical user visualization tasks.
    The dataset is built exclusively from Matplotlib gallery scripts (Section II-A), which may not reflect the diversity or messiness of real-world plotting requests, as the authors note in Section V.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code." pith.science (2026). https://pith.science/paper/DMTQKLKY

@misc{pith2026241202764,
  author       = {Pith},
  title        = {Pith review of: Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMTQKLKY}},
  note         = {Machine review of arXiv:2412.02764}
}
read the original abstract

This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses on generating code for visualizing tabular data - such as a Pandas DataFrame - based on natural language instructions, complementing current evaluation tools and expanding their scope. The dataset includes 175 unique tasks. Our experiments assess several leading Large Language Models (LLMs) across three visualization libraries: Matplotlib, Seaborn, and Plotly. We show that the shortening of tasks has a minimal effect on plotting capabilities, allowing for the user interface that accommodates concise user input without sacrificing functionality or accuracy. Another of our findings reveals that while LLMs perform well with popular libraries like Matplotlib and Seaborn, challenges persist with Plotly, highlighting areas for improvement. We hope that the modular design of our benchmark will broaden the current studies on generating visualizations. Our dataset and benchmark code are available online: https://huggingface.co/datasets/JetBrains-Research/PandasPlotBench; https://github.com/JetBrains-Research/PandasPlotBench.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

    cs.AI 2026-03 conditional novelty 7.0 of 10

    SciVisAgentBench provides 108 expert-crafted tasks and a mixed LLM-plus-deterministic evaluation pipeline for benchmarking AI agents that perform scientific visualization workflows.

  2. MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios

    cs.SE 2025-06 conditional novelty 6.0 of 10

    MLDebugging: a new benchmark of 1,175 multi-library Python debugging tasks on which the best tested LLM, Llama-3.1-72B, passes only 58.7% of test cases.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Le veraging large language models for data analysis automation,

    J. A. Jansen, A. Manukyan, N. Al Khoury, and A. Akalin, “Le veraging large language models for data analysis automation,” PloS one , vol. 20, no. 2, p. e0317084, 2025

    work page 2025

  2. [2]

    Pipe(line) dreams: Fully au tomated end- to-end analysis and visualization,

    C. Beasley and A. Abouzied, “Pipe(line) dreams: Fully au tomated end- to-end analysis and visualization,” in Proceedings of the 2024 W orkshop on Human-In-the-Loop Data Analytics , 2024, pp. 1–7

    work page 2024

  3. [3]

    LLMs f or science: Usage for code generation and data analysis,

    M. Nejjar, L. Zacharias, F. Stiehle, and I. Weber, “LLMs f or science: Usage for code generation and data analysis,” Journal of Software: Evolution and Process , p. e2723, 2023

    work page 2023

  4. [4]

    MatPlotAgent: Method and evaluation for LLM-based agentic scientific data visualization,

    Z. Y ang, Z. Zhou, S. Wang, X. Cong, X. Han, Y . Y an, Z. Liu, Z. Tan, P . Liu, D. Y u et al. , “MatPlotAgent: Method and evaluation for LLM-based agentic scientific data visualization,” arXiv preprint arXiv:2402.11453, 2024

    arXiv 2024

  5. [5]

    Expectat ion vs. experi- ence: Evaluating the usability of code generation tools pow ered by large language models,

    P . V aithilingam, T. Zhang, and E. L. Glassman, “Expectat ion vs. experi- ence: Evaluating the usability of code generation tools pow ered by large language models,” in CHI conference on human factors in computing systems extended abstracts , 2022, pp. 1–7

    work page 2022

  6. [6]

    DS-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.- t. Yih, D. Fried, S. Wang, and T. Y u, “DS-1000: A natural and reliable benchmark for data science code generation,” in International Conference on Machine Learning . PMLR, 2023, pp. 18 319–18 345

    work page 2023

  7. [7]

    Plot2Code: A comprehensive benchmark for evaluating mult i-modal large language models in code generation from scientific plo ts,

    C. Wu, Y . Ge, Q. Guo, J. Wang, Z. Liang, Z. Lu, Y . Shan, and P . Luo, “Plot2Code: A comprehensive benchmark for evaluating mult i-modal large language models in code generation from scientific plo ts,” arXiv preprint arXiv:2405.07990, 2024

    arXiv 2024

  8. [8]

    ChartMimic: Evaluating LMM’s cross- modal reasoning capability via chart-to-code generation,

    C. Shi, C. Y ang, Y . Liu, B. Shui, J. Wang, M. Jing, L. Xu, X. Zhu, S. Li, Y . Zhang et al. , “ChartMimic: Evaluating LMM’s cross- modal reasoning capability via chart-to-code generation, ” arXiv preprint arXiv:2406.09961, 2024

    arXiv 2024

Show all 24 references
  1. [9]

    nvBench: A large-scale synthe sized dataset for cross-domain natural language to visualization task,

    Y . Luo, J. Tang, and G. Li, “nvBench: A large-scale synthe sized dataset for cross-domain natural language to visualization task,” arXiv preprint arXiv:2112.12926, 2021

    2021 arXiv

  2. [10]

    pandas-dev/pandas: Pan das,

    The pandas development team, “pandas-dev/pandas: Pan das,” 2020. [Online]. Available: https://doi.org/10.5281/zenodo.3 509134

    2020 doi

  3. [11]

    Hello GPT-4o,

    OpenAI, “Hello GPT-4o,” 2024, accessed: 2025-02-10. [ Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  4. [12]

    The Claude 3 model family: Opus, Sonnet, Ha iku,

    Anthropic, “The Claude 3 model family: Opus, Sonnet, Ha iku,” 2024, accessed: 2025-02-10. [Online]. Available: https://www. anthropic.com

    2024

  5. [13]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P . Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gul ati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  6. [14]

    The Llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A . Letman, A. Mathur, A. Schelten, A. Y ang, A. Fan et al. , “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  7. [15]

    Matplotlib: A 2D graphics environment,

    J. D. Hunter, “Matplotlib: A 2D graphics environment,” Computing in science & engineering , vol. 9, no. 03, pp. 90–95, 2007

    2007

  8. [16]

    Seaborn: statistical data visualizatio n,

    M. L. Waskom, “Seaborn: statistical data visualizatio n,” Journal of Open Source Software, vol. 6, no. 60, p. 3021, 2021

    2021

  9. [17]

    Collaborative data scienc e,

    Plotly Technologies Inc., “Collaborative data scienc e,” Montreal, QC, 2015, accessed: 2025-02-10. [Online]. Available: https:/ /plot.ly

    2015

  10. [18]

    HuggingFace Page with PandasPlotBench,

    T. Galimzyanov, S. Titov, Y . Golubev, and E. Bogomolov, “HuggingFace Page with PandasPlotBench,” 2024, accessed: 2025-02-10. [Online]. Available: https://huggingface.co/datasets/ JetBrains-Research/ PandasPlotBench

    2024

  11. [19]

    Code for running the benchmark,

    ——, “Code for running the benchmark,” 2024, accessed: 2 025- 02-10. [Online]. Available: https://github.com/JetBrai ns-Research/ PandasPlotBench

    2024

  12. [20]

    Supplementary materials,

    ——, “Supplementary materials,” 2024, accessed: 2025- 02-10. [Online]. Available: https://github.com/JetBrains-Research/Pan dasPlotBench/tree/ main/paper_supp_info

    2024

  13. [21]

    MatPlotLib Gallery,

    “MatPlotLib Gallery,” accessed: 2025-02-10. [Online ]. Available: https://matplotlib.org/stable/gallery/index.html

    2025

  14. [22]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L . Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  15. [23]

    GPT-4V(ision) System Card,

    OpenAI, “GPT-4V(ision) System Card,” 2023, accessed: 2025-02-10. [Online]. Available: https://cdn.openai.com/papers/GP TV_System_Card. pdf

    2023

  16. [24]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou , B. Qin, T. Liu, D. Jiang et al. , “CodeBERT: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

    2002 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.