Automated Visualization Makeovers with LLMs

David A. Selby; Sebastian J. Vollmer; Siddharth Gangwar

REVIEW 4 major objections 5 minor 1 cited by

Automated Visualization Makeovers with LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A modular prompting system lets a multimodal LLM critique charts from image or code, and the paper shows it catches objective structural errors like non-zero baselines and dual axes with perfect F1-scores, while stylistic issues like…

desk verdict Worth refereeing: a useful modular chart-critique system whose n=6 evaluation is too thin to support 'reliable'—resubmit with statistics and a baseline. read the letter →

arxiv 2508.05637 v1 pith:LTR73R2C submitted 2025-07-21 cs.HC cs.AI

Siddharth Gangwar , David A. Selby , Sebastian J. Vollmer This is my paper · ORCID

classification cs.HCcs.AI

keywords largelanguagemodelsdatavisualizationpromptengineeringmisleadingchartsautomatedcritiquecharttypedetectionbestpracticesmultimodalLLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a modular, chart-type-specific prompting system can make a multimodal large language model a useful semi-automatic critic of existing data visualizations. The system accepts a chart as an image or as code, detects the chart type, loads design rules written for that type, and returns natural-language feedback plus optional corrected code. The authors argue that their quantitative evaluation of 72 synthetic chart images shows the approach is reliable for objective structural errors, with perfect F1-scores for non-zero baselines and dual-axis issues, while stylistic issues such as inappropriate colour choices remain hard. If true, this matters because many practitioners lack formal training in visualization design, and an automated critic could serve as an educational assistant and a pre-publication linter. The paper is explicit that its curated synthetic dataset limits generalization to real-world inputs.

What carries the argument

The central mechanism is a modular, multi-stage prompting pipeline: chart-type detection, threshold evaluation of chart properties, loading of chart-type-specific rules from a structured JSON rule file, issue detection with natural-language feedback, optional code correction when the input is code, and display of results. The rule sets codify best practices such as 'no more than 7 pie slices' and 'avoid dual axes for line charts'; the LLM performs the visual reasoning while the rules supply context and constrain the critique.

What would settle it

Run the identical modular prompting system on a corpus of real-world charts with independent human annotation of the same 12 error types; if the F1-scores for non-zero baselines and dual-axis issues drop noticeably from 1.00, or the mean absolute error in issue count rises well above 0.44, the central claim of reliable structural-error detection in realistic settings is refuted.

Watch

Extended reading notes

Core claim

The paper claims that a multimodal LLM, guided by a modular prompting pipeline that first detects the chart type, then loads chart-specific design rules and thresholds, can act as a semi-automatic visualization critic: given an image or code input, it identifies design violations, explains them, and optionally returns corrected code. In a synthetic evaluation of 72 images across 12 error types, the system attained perfect F1-scores (1.00) for Non-Zero Baselines and Dual Axis Issues, high scores for Too Many Slices, Improper Axis Scaling, and Inconsistent Bar Widths, and markedly lower scores for Inappropriate Colour Choices (0.46) and Overlapping Data Elements (0.63); the mean absolute error in the predicted number of issues was 0.44, with a slight tendency to underestimate. The authors conclude that objective structural errors can be caught reliably while stylistic issues remain more ambiguous.

Load-bearing premise

The load-bearing premise is that the 72-image synthetic dataset with manually curated labels is representative enough that the measured scores carry over to real, uncontrolled charts; the authors concede this is a limitation.

Editorial extensions

If this is right

The system can function as an automated pre-publication linter for charts, catching objective structural errors before a graphic reaches an audience.
For code-based inputs, the optional corrected-code output turns the tool into an educational aid that teaches better practice rather than just flagging problems.
The strong contrast in F1-scores between structural and stylistic errors indicates that unambiguous, rule-codifiable checks are the main driver of success, while visual ambiguity limits performance.
The mean absolute error of 0.44 and slight underestimation mean that on complex charts with overlapping issues, users should expect some errors to be missed or conflated.
Because the system accepts both image and code inputs, it can critique charts produced across scripting libraries and GUI-based tools alike.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because the rule files are external JSON, the system is effectively an editable linter; a user or organization could codify its own house style and enforce those preferences without retraining.
Beyond the paper: the structural-versus-stylistic performance gap suggests a division of labour in which rule-codifiable checks can be automated today, while subjective aesthetic judgments will need user preferences, larger human-rated corpora, or learned models.
Beyond the paper: the same modular chart-type detection and rule-loading pipeline could transfer to other visual artefacts such as maps, dashboards, or infographics, where analogous grammatical and stylistic error categories exist.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing: this is a genuine system paper, not a world-changer. The contribution is modular, chart-type-specific prompting for a multimodal LLM (GPT-4o) to critique visualizations from either images or code, plus a small labeled synthetic dataset and a working web app. That combination is new relative to Lo and Qu, which used general prompts for misleading-chart detection. The paper is clearly written and honest: the limitations about generalizability are stated in Section 7, and no fitted parameters are hidden anywhere.

What it does well: the rule sets are sensible and loaded per chart type; supporting both code and image input is practically useful; the evaluation is structured around per-error F1 and MAE, which is appropriate; and the metric numbers, though fragile, are computed correctly as far as I can tell. The dataset is on OSF, so reproducibility is possible. The authors don't overclaim about correcting charts; they focus on error detection.

The soft spot is exactly what the stress-test flags: n=6 per error type and a single non-deterministic LLM run. With six positive examples, the perfect F1 for Non-Zero Baselines and Dual Axis Issues has a binomial 95% CI for recall that's roughly 54–100%. One mistake changes the F1 by about 0.17. There are no repeated runs, no confidence intervals, and no baseline (how would general-prompt GPT-4o or a simple rule-based linter do?). So the phrase 'performs reliably' in Section 6 is stronger than the evidence supports. This is not a fatal flaw; it's a sizing problem. The paper would be credible with a modest real-world sample or at least repeated sampling and error bars.

Also minor: some thresholds ('no more than 7 pie slices') are free parameters, but they are explicit and conventional, so I don't see that as circular. The MAE numbers would benefit from a per-subset breakdown, though they do show single- vs multi-error.

Who is this for? People building LLM-based critique tools for data visualization, and anyone studying how to evaluate such systems. It deserves a serious referee—I would not desk-reject it. But I would ask the authors to strengthen the statistics and add a baseline before publication. With that, it's a solid short paper.

Referee Report

4 major / 5 minor

Summary. The paper presents a multimodal LLM-based system for automated visualization critique. Given a chart image or the code that generated it, the system detects the chart type, loads chart-specific design rules from a JSON file, and prompts GPT-4o to identify violations such as non-zero baselines, dual-axis issues, or excessive pie slices, returning natural-language feedback and (for code inputs) corrected code. The authors evaluate the system on a synthetic dataset of 72 chart images spanning 12 error types, reporting per-error-type precision, recall, and F1, plus a mean absolute error (MAE) of 0.44 for the predicted number of issues. They conclude that the system performs reliably on objective structural errors while struggling with stylistic or ambiguous issues.

Significance. If the reported results are robust, the system is a useful educational tool for visualization makeovers and a nice demonstration of combining structured, chart-specific rules with the visual reasoning of a multimodal LLM. The paper's strengths include the explicit, modular prompt architecture; the release of the dataset on OSF; and the use of standard multi-label classification metrics. However, the evidence currently falls short of establishing the headline claim: the per-class sample size is just six images, each processed once, with no confidence intervals, no repeated runs, and no baseline or ablation comparison. These are not merely presentation issues; they directly affect whether the central claim of 'reliable' structural error detection is supported.

major comments (4)

[Section 5 (Dataset) and Section 6 (Table 2)] The perfect F1 scores for Non-Zero Baselines and Dual Axis Issues are computed from only six positive examples per class, with each image processed exactly once by a non-deterministic LLM. With n=6, a single misclassification changes F1 by roughly 0.17, and even a recall of 6/6 carries a wide binomial 95% confidence interval (approximately 0.54-1.00). The paper should report confidence intervals via bootstrap or repeated sampling (e.g., multiple runs with varied temperature), or substantially increase the number of examples per error type. As written, the claim that the system 'performs reliably on objective structural errors' is underdetermined by the evaluation.
[Section 4 (Implementation) vs. Section 6 (Results)] No baseline or ablation is reported. The contribution is a modular, chart-type-specific prompting pipeline, yet there is no comparison with a generic prompt that asks the same model to detect all 12 issues without chart-specific rules, nor with a non-modular rule set, nor with prior work such as Lo and Qu's evaluation of LLMs on misleading chart detection. Without such a comparison, the measured performance cannot be attributed to the proposed framework, and the claimed advantage over general prompting remains unsupported.
[Section 6 (Metrics and MAE)] The overall MAE of 0.44 is a point estimate from a single run over 72 images, with no measure of variance, confidence interval, or significance test. The comparison between single-error (0.37) and multi-error (0.51) subsets likewise lacks uncertainty quantification, and the difference is not shown to be meaningful at this sample size. The paper should provide bootstrap intervals or repeated-run statistics for the MAE and for the single/multi-error split.
[Section 7 (Conclusion & Future Work)] The authors correctly acknowledge that the evaluation relies on a curated, semi-synthetic dataset, but the central conclusion should be tempered accordingly in the abstract and introduction. Because the synthetic charts are created to exhibit the same error taxonomy that is encoded in the system's rule sets, the reported F1 scores measure rule-following on expected stimuli rather than independent visual-communication quality. Including at least a small set of real-world charts with qualitative analysis would help determine whether the 'objective structural error' finding generalizes beyond the controlled setting.

minor comments (5)

[Section 8 (Acknowledgments)] Typo: 'provded' should be 'provided'.
[Author affiliations] The second author's name appears as 'David A. Selby' in the header but 'David_antony.selby' in the footnote; the email formatting and the spacing in 'S. J. V ollmer' should be corrected.
[Section 4 (Implementation, Figure 2 description)] The text 'non-zeroy-axis baseline' should be 'non-zero y-axis baseline'.
[Section 1 (Introduction)] The term '3-d effects' appears in the introduction; for consistency use '3D effects' as in the evaluation section.
[Section 5 (Dataset)] The paper states 'Roughly 42% of the dataset (30 images) was designed to contain more than one error'; 30/72 is 41.7%, and the remaining 42 images are single-error. Please ensure the percentages and counts are reported consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation measures GPT-4o's compliance with explicit, externally-sourced rule sets against manually annotated ground-truth labels; no fitted parameters or self-citations are involved.

full rationale

The paper's derivation chain is fully self-contained and non-circular. The claimed results (Section 6, Table 2) are measurements of a pre-trained LLM (GPT-4o) applying explicit, chart-type-specific rules stored in a JSON file; those rules are cited to external visualization sources (Tableau documentation [8], Datawrapper [14]) rather than derived from model outputs. Ground-truth labels (Section 5) were manually annotated for 72 synthetically generated charts, independently of the system's predictions. The evaluation is therefore a standard compliance test: does an independent model correctly apply a fixed rule set? That the model fails badly on subjective classes (Inappropriate Colour Choices F1 = 0.46, Overlapping Data Elements F1 = 0.63) demonstrates the results are not forced by construction. The perfect F1 scores for Non-Zero Baselines and Dual Axis Issues are computed from 6 positive examples each in a single run, which limits statistical confidence but is an evidence-strength concern, not circularity. The MAE (0.44) is a descriptive statistic over the same held-out set, not a fitted parameter renamed as a prediction. No parameter is fit to the test set, and no claim is justified by self-citation: the reference list contains no works by the present authors. The Section 7 limitation ('the evaluation relies on a curated, semi-synthetic dataset, which limits generalizability to uncontrolled, real-world inputs') concedes external-validity limits rather than circularity. The taxonomy shared between the rule set and the annotation scheme is standard benchmark design, since the LLM never sees the labels.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The system rests on domain assumptions about LLM visual reasoning and the validity of the hand-written rule set. No new physical or mathematical entities are introduced, and the only free parameters are design thresholds in the rules, which are not fitted to the evaluation data.

free parameters (2)

Pie slice count threshold = Seven slices (rule embedded in JSON prompt)
The prompt enforces 'No more than 7 pie slices'; this hand-chosen threshold is part of the rule set, not derived from data.
Other chart-specific thresholds = Not stated in paper
Stage 2 of the prompt compares properties to 'predefined thresholds' but exact values are not given, so replication requires guessing or adopting the authors' choices.

assumptions (3)

domain assumption GPT-4o's visual perception can reliably read chart elements from bitmap images and apply textual rules to them.
The entire pipeline in Section 4 depends on the multimodal model correctly identifying chart type, axis range, number of slices, etc., which is not independently verified.
domain assumption The 12 error types and their labels in the synthetic dataset cover the most important real-world visualization issues.
Section 5 defines the dataset; the taxonomy is chosen by the authors and the paper notes it is semi-synthetic, limiting generalizability.
domain assumption The predefined best-practice rules (avoid dual axes, zero baseline, etc.) are valid and uncontroversial.
Rules are loaded from a static JSON file in Section 4; their correctness as universal design guidance is not empirically tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Visualization Makeovers with LLMs." pith.science (2026). https://pith.science/paper/LTR73R2C

@misc{pith2026250805637,
  author       = {Pith},
  title        = {Pith review of: Automated Visualization Makeovers with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTR73R2C}},
  note         = {Machine review of arXiv:2508.05637}
}

read the original abstract

Making a good graphic that accurately and efficiently conveys the desired message to the audience is both an art and a science, typically not taught in the data science curriculum. Visualisation makeovers are exercises where the community exchange feedback to improve charts and data visualizations. Can multi-modal large language models (LLMs) emulate this task? Given a plot in the form of an image file, or the code used to generate it, an LLM, primed with a list of visualization best practices, is employed to semi-automatically generate constructive criticism to produce a better plot. Our system is centred around prompt engineering of a pre-trained model, relying on a combination of userspecified guidelines and any latent knowledge of data visualization practices that might lie within an LLMs training corpus. Unlike other works, the focus is not on generating valid visualization scripts from raw data or prompts, but on educating the user how to improve their existing data visualizations according to an interpretation of best practices. A quantitative evaluation is performed to measure the sensitivity of the LLM agent to various plotting issues across different chart types. We make the tool available as a simple self-hosted applet with an accessible Web interface.

Figures

Figures reproduced from arXiv: 2508.05637 by the authors.

**Figure 2.** Demonstration: Uploaded chart (sourced from [14]) is analyzed by the system. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Structured Visualization Design Knowledge for Grounding Generative Reasoning and Situated Feedback
cs.HC 2025-12 conditional novelty 6.0 of 10

A role-annotated guideline scheme makes visualization design knowledge machine-queryable and source-traceable, demonstrated with a 744-guideline catalog and embedding-based analyses.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages · cited by 1 Pith paper

[1]

B. Bach, M. Keck, F. Rajabiyazdi, T. Losev, I. Meirelles, J. Dykes, R. S. Laramee, M. AlKadi, C. Stoiber, S. Huron, C. Perin, L. Morais, W. Aigner, D. Kosminsky, M. Boucher, S. Knud- sen, A. Manataki, J. Aerts, U. Hinrichs, J. C. Roberts, and S. Carpendale. Challenges and opportunities in data visualization education: A call to action. IEEE Transactions o...

arXiv 2024
[3]

Q. Chen, F. Sun, X. Xu, Z. Chen, J. Wang, and N. Cao. Vizlinter: A linter and fixer frame- work for data visualization. IEEE Transactions on Visualization and Computer Graphics , 28(1):206–216, 2022. doi: 10.1109/TVCG.2021.3114804

arXiv 2022
[4]

J. Choi, J. Lee, and J. Jo. Bavisitter: Integrating design guidelines into large language models for visualization authoring. In 2024 IEEE Visualization and Visual Analytics (VIS), pp. 121– 125, 2024. doi: 10.1109/VIS55277.2024.00032

arXiv 2024
[5]

V . Dibia. LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. In D. Bollegala, R. Huang, and A. Ritter, eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 3: System Demonstrations), pp. 113–126. Association for Computational Linguis- tic...

doi:10.18653/v1/2023.acl-demo.11 2023
[6]

Hopkins, M

A. Hopkins, M. Correll, and A. Satyanarayan. VisuaLint: Sketchy in situ annotations of chart construction errors. In Proceedings of the Eurographics Conference on Visualization (EuroVis), vol. 29, July 2020. doi: 10.1111/cgf.13975

doi:10.1111/cgf.13975 2020
[7]

N. W. Kim, G. Myers, and B. Bach. How good is ChatGPT in giving advice on your visual- ization design?, Apr. 2024. doi: 10.48550/arXiv.2310.09617

arXiv doi:10.48550/arxiv.2310.09617 2024
[8]

S. Kraemer. How to spot misleading charts: Check the axes. https://www.tableau.com/ blog/how-spot-misleading-charts-check-axes , 2024

work page 2024
[9]

Kriebel and E

A. Kriebel and E. Murray. #MakeoverMonday: Improving how we visualize and analyze data, one chart at a time . John Wiley & Sons, Oct. 2018

work page 2018

Show all 16 references

[10]

Liew and K

A. Liew and K. Mueller. Using large language models to generate engaging captions for data visualizations, Dec. 2022. doi: 10.48550/arXiv.2212.14047

2022 doi arXiv
[11]

L. Y .-H. Lo, Y . Cao, L. Yang, and H. Qu. Why change my design: Explaining poorly con- structed visualization designs with explorable explanations. IEEE Transactions on Visual- ization and Computer Graphics , 30(1):955–964, Jan. 2024. doi: 10.1109/TVCG.2023.3327155

2024
[12]

L. Y .-H. Lo, A. Gupta, K. Shigyo, A. Wu, E. Bertini, and H. Qu. Misinformed by visual- ization: What do we learn from misinformative visualizations? Computer Graphics F orum, 41(3):515–525, Aug. 2022. doi: 10.1111/cgf.14559

2022 doi
[13]

L. Y .-H. Lo and H. Qu. How good (or bad) are LLMs at detecting misleading visualizations? IEEE Transactions on Visualization and Computer Graphics , 31(1):1116–1125, Jan. 2025. doi: 10.1109/TVCG.2024.3456333

2025
[14]

L. C. Muth. Why not to use two axes, and what to use instead. https://www. datawrapper.de/blog/dualaxis, 2018

2018
[15]

Achiam, S

OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Bal- tescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner,...

2024
[16]

L. Shen, E. Shen, Y . Luo, X. Yang, X. Hu, X. Zhang, Z. Tai, and J. Wang. Towards natural language interfaces for data visualization: A survey.IEEE Transactions on Visualization and Computer Graphics, 29(6):3121–3144, June 2023. doi: 10.1109/TVCG.2022.3148007

2023
[17]

G. Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. doi: 10.48550/arXiv.2403.05530

2024 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

B. Bach, M. Keck, F. Rajabiyazdi, T. Losev, I. Meirelles, J. Dykes, R. S. Laramee, M. AlKadi, C. Stoiber, S. Huron, C. Perin, L. Morais, W. Aigner, D. Kosminsky, M. Boucher, S. Knud- sen, A. Manataki, J. Aerts, U. Hinrichs, J. C. Roberts, and S. Carpendale. Challenges and opportunities in data visualization education: A call to action. IEEE Transactions o...

arXiv 2024

[2] [3]

Q. Chen, F. Sun, X. Xu, Z. Chen, J. Wang, and N. Cao. Vizlinter: A linter and fixer frame- work for data visualization. IEEE Transactions on Visualization and Computer Graphics , 28(1):206–216, 2022. doi: 10.1109/TVCG.2021.3114804

arXiv 2022

[3] [4]

J. Choi, J. Lee, and J. Jo. Bavisitter: Integrating design guidelines into large language models for visualization authoring. In 2024 IEEE Visualization and Visual Analytics (VIS), pp. 121– 125, 2024. doi: 10.1109/VIS55277.2024.00032

arXiv 2024

[4] [5]

V . Dibia. LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. In D. Bollegala, R. Huang, and A. Ritter, eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 3: System Demonstrations), pp. 113–126. Association for Computational Linguis- tic...

doi:10.18653/v1/2023.acl-demo.11 2023

[5] [6]

Hopkins, M

A. Hopkins, M. Correll, and A. Satyanarayan. VisuaLint: Sketchy in situ annotations of chart construction errors. In Proceedings of the Eurographics Conference on Visualization (EuroVis), vol. 29, July 2020. doi: 10.1111/cgf.13975

doi:10.1111/cgf.13975 2020

[6] [7]

N. W. Kim, G. Myers, and B. Bach. How good is ChatGPT in giving advice on your visual- ization design?, Apr. 2024. doi: 10.48550/arXiv.2310.09617

arXiv doi:10.48550/arxiv.2310.09617 2024

[7] [8]

S. Kraemer. How to spot misleading charts: Check the axes. https://www.tableau.com/ blog/how-spot-misleading-charts-check-axes , 2024

work page 2024

[8] [9]

Kriebel and E

A. Kriebel and E. Murray. #MakeoverMonday: Improving how we visualize and analyze data, one chart at a time . John Wiley & Sons, Oct. 2018

work page 2018