Pith. sign in

REVIEW 3 major objections 6 minor 55 references

Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large language model agents can rate visualizations like human users, but this paper shows their agreement with human conclusions is largely limited to experiments where expert hypotheses are confident.

desk verdict A useful empirical study of LLM agents as rating proxies, but the central expert-confidence correlation is a marginal post-hoc pattern—closer to a hypothesis than a demonstration. read the letter →

arxiv 2505.06702 v1 pith:KOSSBM6P submitted 2025-05-10 cs.HC

Zekai Shao , Yi Shan , Yixuan He , Yuxuan Yao , Junhong Wang , Xiaolong (Luke) Zhang , Yu Zhang , Siming Chen This is my paper · ORCID
classification cs.HC
keywords largelanguagemodelsvisualizationevaluationhuman-agentalignmentuserstudysimulationsubjectiveratingsexpertconfidenceretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model agent can take the place of human participants in rating visualizations, and it answers with a conditional yes. Across three studies, the authors replicated six published human-subject experiments by feeding the original stimuli and instructions to a multimodal language model that rated the charts on confidence, ease of use, readability, or aesthetics. The central finding is that the agent's ratings reproduce the human conclusion when external visualization experts were able to form consistent, high-confidence hypotheses about the experiment; for moderate-confidence or hypothesis-free questions, alignment drops sharply or disappears. The paper also finds that common enhancement techniques—changing image inputs, wording comparisons explicitly, or injecting web-retrieved knowledge—can shift agent ratings, but can also inject new biases. The bottom line, stated by the authors, is that agent simulation can complement and inform user studies but cannot replace them.

What carries the argument

The argument is carried by a comparison table in which every conclusion from the six replications carries three labels: whether the original study's hypothesis (H) matched its conclusion (C), whether the agent feedback (A) matched the hypothesis, and whether it matched the conclusion. Onto these rows the paper grafts a confidence score per conclusion, obtained by asking five external visualization experts to form their own predictions for each experiment and rate their certainty on a three-point scale, with opposed expert hypotheses downgraded to low; the five scores are then averaged and rounded. The agent runs themselves use a fixed replication protocol—the original instructions and stimuli fed to a multimodal language model, with between-subject designs converted to within-subject batches because the model cannot hold a stable rating standard across separate sessions.

What would settle it

Take the six replicated experiments and have two independent coders re-label, from the raw transcripts, whether agent feedback matches each hypothesis and each conclusion, then recompute the match-rate difference between high- and low-confidence rows; if inter-coder agreement is low or the gap disappears, the paper's central correlation does not hold. A complementary test would pre-register expert hypotheses and confidence before a new set of visualization user studies and check whether agent-human agreement rises monotonically with pre-registered confidence.

Watch

Extended reading notes

Core claim

The paper claims that alignment between language-model agents and human raters in visualization experiments is predictable from expert confidence. In six replications, the match rate between agent and human conclusions was 3 out of 5 for high-confidence expert hypotheses, but 1 out of 12 elsewhere; the one high-confidence case that failed was a magnitude-judgement task where the agent consistently inverted the human interpretation by reading axis labels as text. The authors interpret this pattern as evidence that the agent behaves like an aggregator of historically trained knowledge: it can reproduce basic, well-established perceptual findings, but it does not possess the visual and cognitive machinery to track the subtle or contested effects that human experiments are usually run to discover.

Load-bearing premise

The correlation between expert confidence and agent alignment is only as strong as the hand-assigned labels in the comparison table and the experts' self-reported confidence; if those labels or confidence ratings are unreliable, the correlation may be an artifact.

Editorial extensions

If this is right

  • When experts have low confidence in a hypothesis, agent ratings are unlikely to reproduce the human conclusion, so low-confidence conditions still require human participants.
  • For basic, high-confidence findings—noise changes perceived fit, horizontal timelines are most readable—an agent can serve as a quick pre-check before a small pilot study.
  • Prompt and input choices are consequential: explicitly comparing variables or removing images pushes the agent toward textual stereotypes, so agent-based evaluations must report and validate their exact prompt configuration.
  • Web-retrieved knowledge injection can repair a specific reasoning error, such as aggregating icicle children, but the retrieved knowledge is uncertain in relevance and cannot yet be treated as a general fix.
  • If an agent already matches a validated user study, changing only experimental parameters, such as a more extreme decentering condition, can provide a preliminary preview of the new outcome and speed up iterative design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the replication protocol converted between-subject designs into within-subject batches, the reported alignment may be an upper bound for how an agent would do when each condition is judged alone, as in most field use.
  • The high-confidence alignments are consistent with the agent reproducing patterns it has seen during training; a decisive test would run the same protocol on experiments published after the model's knowledge cutoff.
  • The five-expert confidence coding suggests a practical gate: pre-register expert confidence for a new visualization question and only trust agent simulation where confidence is high.
  • The failure to model demographic profiles and aesthetic/texture judgments implies agent ratings describe an average reader, not a population; studies in which individual differences drive ratings are the least suitable for simulation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether GPT-4V-based agents can simulate human ratings in visualization experiments across three studies. Study I replicates a published CHI time-series user study and finds that agent ratings can mimic human-like reasoning but fail to capture user diversity. Study II replicates six human-subject experiments from five OSF-sourced papers, introduces external expert confidence coding, and reports that agent-human alignment positively correlates with experts' pre-experiment confidence. Study III tests input preprocessing and knowledge injection, finding that these techniques can improve or bias agent ratings. The paper concludes that agents can potentially simulate human ratings when guided by high-confidence expert hypotheses, while explicitly cautioning that agents cannot replace user studies.

Significance. If the central claim holds, the paper makes a useful contribution to visualization evaluation methodology by identifying when LLM agents can serve as low-cost proxies in formative rating studies. The work uses external published human studies as benchmarks, avoiding circular validation, and the authors provide open code, detailed replication protocols, and candid discussions of limitations such as data pollution and modified experimental procedures. The proposed fast-prototyping scenario in Sec. 6 is a concrete, falsifiable use case. However, the significance is bounded by the small dataset (five papers, six experiments) and the fragility of the headline confidence-alignment correlation, which is supported by a small non-independent sample and coding procedures that need better verification.

major comments (3)
  1. [Sec. 4.6, Table 1] The central claim that agent-human alignment positively correlates with expert confidence is supported by a single 2x2 split: 3/5 match in high-confidence rows versus 1/12 elsewhere. This split is not statistically significant (one-sided Fisher exact test yields p about 0.053, and a two-sided test is larger), and the rows are not independent because C1.1, C3.1, and C4.2 are duplicated for different rating types, while all rows come from only five papers. The abstract and conclusion state that the study 'demonstrates' this correlation; given the evidence, this is an overstatement. Please report an analysis that accounts for clustering by paper, or explicitly reframe the finding as exploratory and hypothesis-generating.
  2. [Sec. 4.6, Confidence Coding] The expert confidence judgments were elicited after the agent experiments, and the paper does not describe any blinding of experts to the published outcomes, so the label 'pre-experiment' is asserted rather than verified. Furthermore, the protocol of lowering confidence to 'low' for experts who proposed opposing hypotheses is a post hoc recoding that can inflate the apparent agreement between high confidence and agent-human alignment. In addition, the Y/N/P alignment labels in Table 1 come from a single coding pass without inter-rater reliability. Please provide the full elicitation protocol, blinding details, inter-rater reliability statistics, or at minimum a sensitivity analysis that treats the recoding alternative.
  3. [Sec. 7, Data pollution; Sec. 4.6] The data-pollution discussion in Sec. 7 acknowledges that GPT-4V's training data may inflate alignment for broad, well-known findings, but the same confound is not applied to the expert-confidence explanation. The three high-confidence matches (C1.1, C2.1, C4.1) are broad basic findings ('noise affects fit', 'horizontal timelines are preferred') that are likely overrepresented both in LLM training data and in expert prior knowledge, while the low-confidence rows tend to be detailed pattern claims. The paper does not distinguish the expert-confidence account from a training-data-familiarity account; this needs to be addressed in the revised discussion and ideally in the analysis.
minor comments (6)
  1. [Appendix 1.1, Prompt 3] The prompt text contains a typo: 'answe' should be 'answer'.
  2. [Appendix 1.2] The code block for the Imputation for Uncertainty execution is duplicated verbatim; one copy should be removed.
  3. [Appendix 2.5 and Sec. 5.2] The model name is written inconsistently as 'GPT-4V' and 'GPT-4v'; please standardize to 'GPT-4V' throughout.
  4. [Table 1] Several rows (C2.4, C4.3, C5.1, C5.2, C6.2) have blank entries under H-C, H-A, or C-A; please clarify whether these denote 'no hypothesis' or 'not applicable' in the table caption.
  5. [Abstract and Sec. 4.1] The abstract says the second study 'repeated six human-subject studies' while Sec. 4.1 states that 'we identified five suitable papers'; please clarify that six experiments were drawn from five papers.
  6. [Sec. 4.6, Implications] The sentence 'Our response to RQ2 suggests...' should read 'Our answer to RQ2 suggests...' for consistency with the RQ phrasing used elsewhere.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the comparison against published human studies is self-contained.

full rationale

The paper's core comparison is anchored to external, published human-subject studies (Adnan et al. 2016; Reimann et al. 2021; Sarma et al. 2023; Di Bartolomeo et al. 2020; He et al. 2024; Bradley et al. 2025) with open-source stimuli and human ratings. Agent ratings are generated from those fixed stimuli and prompts, and alignment is scored against the externally fixed human conclusions. No equation in the paper fits a parameter to the target outcome; the 3/5 versus 1/12 summary in Sec. 4.6 is a post-hoc descriptive comparison rather than a fitted prediction. The related-work self-citations (e.g., refs. 12, 41, 46, 56) are background context and are not load-bearing for the central claim. The weakest point is the confidence-coding procedure in Sec. 4.6: the authors consulted experts after the fact, recoded opposing expert hypotheses to low confidence, and the Table 1 Y/N/P alignment labels come from a single coding pass. These are legitimate threats to the independence and validity of the claimed correlation, and the paper's own Sec. 7 data-pollution discussion reinforces that the correlation may be confounded with task triviality or training-data overlap. However, confounding and weak statistical support are not constructional circularity: the predictor (expert confidence) and the outcome (agent-human conclusion alignment) are not the same variable by definition, and the claimed correlation is an empirical, testable observation rather than a result forced by the paper's definitions. There is no demonstrated step in which a prediction is equivalent to its input by construction, and no load-bearing self-citation chain. Therefore the appropriate circularity finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No mathematical free parameters are fitted. The analysis rests on four domain assumptions: conclusion-level comparison is a valid alignment proxy; the filtered OSF papers represent visualization rating studies; five experts' post hoc confidence codes are independent; and GPT-4V represents LLM agents. No invented entities are introduced.

assumptions (4)
  • domain assumption Conclusion-level comparison is a valid proxy for alignment; absolute rating baselines can be ignored if conclusions match.
    Sec. 4.2.2 says the agent's rating baseline differs from humans and the problem is resolved by comparing conclusions rather than absolute values. If scale compression or baseline shifts distort conclusion direction, the alignment measurement fails.
  • domain assumption The five selected OSF papers adequately represent visualization evaluation studies with subjective ratings.
    Sec. 4.1 filtering selected five papers; the correlation claim in Sec. 4.6 is derived from this small, self-selected set.
  • domain assumption Five external experts' post hoc confidence ratings are independent and valid measures of pre-experiment confidence.
    Sec. 4.6 says experts were presented materials without original hypotheses, but their knowledge of the literature is not controlled, and opposing hypotheses were recoded to low confidence by the authors.
  • domain assumption GPT-4V is treated as a representative agent for LLM-based rating behavior.
    Sec. 4.1 says GPT-4V was the primary model; generalizability to other LLMs is discussed but not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study." pith.science (2026). https://pith.science/paper/KOSSBM6P

@misc{pith2026250506702,
  author       = {Pith},
  title        = {Pith review of: Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOSSBM6P}},
  note         = {Machine review of arXiv:2505.06702}
}
read the original abstract

Large language models encode knowledge in various domains and demonstrate the ability to understand visualizations. They may also capture visualization design knowledge and potentially help reduce the cost of formative studies. However, it remains a question whether large language models are capable of predicting human feedback on visualizations. To investigate this question, we conducted three studies to examine whether large model-based agents can simulate human ratings in visualization tasks. The first study, replicating a published study involving human subjects, shows agents are promising in conducting human-like reasoning and rating, and its result guides the subsequent experimental design. The second study repeated six human-subject studies reported in literature on subjective ratings, but replacing human participants with agents. Consulting with five human experts, this study demonstrates that the alignment of agent ratings with human ratings positively correlates with the confidence levels of the experts before the experiments. The third study tests commonly used techniques for enhancing agents, including preprocessing visual and textual inputs, and knowledge injection. The results reveal the issues of these techniques in robustness and potential induction of biases. The three studies indicate that language model-based agents can potentially simulate human ratings in visualization experiments, provided that they are guided by high-confidence hypotheses from expert evaluators. Additionally, we demonstrate the usage scenario of swiftly evaluating prototypes with agents. We discuss insights and future directions for evaluating and improving the alignment of agent ratings with human ratings. We note that simulation may only serve as complements and cannot replace user studies.

Figures

Figures reproduced from arXiv: 2505.06702 by the authors.

Figure 2
Figure 2. The data preparation steps. The values represent the num￾ber of remaining papers after each step. For VIS papers, we skip the keyword matching step and utilized vispubdata [22] dataset to obtain all paper titles. scarcely affected the outcomes, suggesting GPT-4V’s limited grasp on how different demographics influence user behavior, thus failing to represent individual differences. Additionally, we adjusted the agent… view at source ↗
Figure 1
Figure 1. Agent ratings for time series visualization tasks. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Results of agent ratings for fit estimations under various [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Proportion of agent ratings ≥ 3 for imputation methods ad￾dressing uncertainty. They feature two tasks (estimating the average and trend) across three conditions: (A) text and images combined, (B) text only with the same prompt, and (C) text only with a general descrip…
Figure 6
Figure 6. Figure 6: Scatterplots: strong decentering (0.99) with 0.2 noise (R1) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Fit ratings by agent (left) and human (right) for different [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Results of GPT-4o’s ratings for fit estimations under various [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 25 canonical work pages

  1. [1]

    Adnan, M

    M. Adnan, M. Just, and L. Baillie. Investigating Time Series Visual- isations to Improve the User Experience. InProceedings of the CHI Conference on Human Factors in Computing Systems, pp. 5444–5455. Association for Computing Machinery, New York, NY , USA, 2016. doi: 10.1145/2858036.2858300 1, 2, 6

    arXiv 2016

  2. [2]

    K. Andrews. Evaluation Comes in Many Guises. InAVI Workshop on BEyond time and errors (BELIV) Position Paper, pp. 7–8, 2008. 2

    work page 2008

  3. [3]

    Barkhuus and J

    L. Barkhuus and J. A. Rode. From Mice to Men - 24 Years of Eval- uation in CHI. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machin- ery, New York, NY , USA, 2007. doi: 10.1145/1240624.2180963 2

    work page arXiv 2007

  4. [4]

    Bradley, G

    D. Bradley, G. Strain, C. Jay, and A. J. Stewart. Magnitude Judgements Are Influenced by the Relative Positions of Data Points Within Axis Limits.IEEE Transactions on Visualization and Com- puter Graphics, 31(2):1414–1421, 2025. doi: 10.1109/TVCG.2024. 3364069 3, 6, 13

    doi:10.1109/tvcg.2024 2025

  5. [5]

    Chen and H

    M. Chen and H. Jäenicke. An Information-theoretic Framework for Visualization.IEEE Transactions on Visualization and Computer Graphics, 16(6):1206–1215, 2010. doi: 10.1109/TVCG.2010.132 2

    doi:10.1109/tvcg.2010.132 2010

  6. [7]

    J. W. Creswell and J. D. Creswell.Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications, 2017. 2

    work page 2017

  7. [9]

    Z. Cui, L. Chen, Y . Wang, D. Haehn, Y . Wang, and H. Pfister. Gener- alization of CNNs on Relational Reasoning with Bar Charts.IEEE Transactions on Visualization and Computer Graphics, pp. 1–15,

    work page

  8. [10]

    Di Bartolomeo, A

    S. Di Bartolomeo, A. Pandey, A. Leventidis, D. Saffo, U. H. Syeda, E. Carstensdottir, M. Seif El-Nasr, M. A. Borkin, and C. Dunne. Eval- uating the Effect of Timeline Shape on Visualization Task Perfor- mance. InProceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–12. Association for Computing Machin- ery, New York, NY , USA, 202...

    arXiv 2020

Show all 55 references
  1. [11]

    L. Fan, W. Hua, X. Li, K. Zhu, M. Jin, L. Li, H. Ling, J. Chi, J. Wang, X. Ma, and Y . Zhang. NPHardEval4V: A Dynamic Reasoning Bench- mark of Multimodal Large Language Models, 2024. doi: 10.48550/ arXiv.2403.01777 6

    2024 doi arXiv

  2. [12]

    L. Gao, J. Lu, Z. Shao, Z. Lin, S. Yue, C. Ieong, Y . Sun, R. J. Za- uner, Z. Wei, and S. Chen. Fine-Tuned Large Language Model for Visualization System: A Study on Self-Regulated Learning in Edu- cation.IEEE Transactions on Visualization and Computer Graphics, 31(1):514–524, ...

    2025

  3. [13]

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, Q. Guo, M. Wang, and H. Wang. Retrieval-Augmented Generation for Large Language Models: A Survey, 2024. 2, 7

    2024

  4. [14]

    L. W. Ge, Y . Cui, and M. Kay. CALVI: Critical Thinking Assess- ment for Literacy in Visualizations. InProceedings of the CHI Con- ference on Human Factors in Computing Systems, pp. 1–18. Associa- tion for Computing Machinery, New York, NY , USA, 2023. doi: 10. 1145/3544548.3581406 2

    2023

  5. [15]

    Grunde-McLaughlin, M

    M. Grunde-McLaughlin, M. S. Lam, R. Krishna, D. S. Weld, and J. Heer. Designing LLM Chains by Adapting Techniques from Crowd- sourcing Workflows, 2023. 2, 9

    2023

  6. [16]

    H. Guo, S. R. Gomez, C. Ziemkiewicz, and D. H. Laidlaw. A Case Study Using Visualization Interaction Logs and Insight Metrics to Un- derstand How Analysts Arrive at Insights.IEEE Transactions on Visu- alization and Computer Graphics, 22(1):51–60, 2016. doi: 10.1109/ TVCG.2015....

    2016

  7. [17]

    Haehn, J

    D. Haehn, J. Tompkin, and H. Pfister. Evaluating ‘graphical percep- tion’ with cnns.IEEE Transactions on Visualization and Computer Graphics, 25(1):641–650, 2019. doi: 10.1109/TVCG.2018.2865138 2

    2019

  8. [19]

    T. He, Y . Zhong, P. Isenberg, and T. Isenberg. Design Characterization for Black-and-White Textures in Visualization.IEEE Transactions on Visualization and Computer Graphics, 30(1):1019–1029, 2024. doi: 10.1109/TVCG.2023.3326941 2, 3, 6, 13

    2024

  9. [20]

    Hullman, X

    J. Hullman, X. Qiao, M. Correll, A. Kale, and M. Kay. In Pursuit of Er- ror: A Survey of Uncertainty Visualization Evaluation.IEEE Transac- tions on Visualization and Computer Graphics, 25(1):903–913, 2019. doi: 10.1109/TVCG.2018.2864889 2

    2019

  10. [21]

    Hämäläinen, M

    P. Hämäläinen, M. Tavast, and A. Kunnari. Evaluating Large Lan- guage Models in Generating Synthetic HCI Research Data: a Case Study. InProceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–19. Association for Computing Machinery, New York, NY , USA, 2...

    2023

  11. [22]

    Isenberg, F

    P. Isenberg, F. Heimerl, S. Koch, T. Isenberg, P. Xu, C. D. Stolper, M. Sedlmair, J. Chen, T. Möller, and J. Stasko. Vispubdata.org: A Metadata Collection About IEEE Visualization (VIS) Publica- tions.IEEE Transactions on Visualization and Computer Graphics, 23(9):2199–2206, 2...

    2017

  12. [23]

    Isenberg, P

    T. Isenberg, P. Isenberg, J. Chen, M. Sedlmair, and T. Möller. A Sys- tematic Review on the Practice of Evaluating Visualization.IEEE Transactions on Visualization and Computer Graphics, 19(12):2818– 2827, 2013. doi: 10.1109/TVCG.2013.126 2

    2013 doi

  13. [24]

    Johansson and C

    J. Johansson and C. Forsell. Evaluation of Parallel Coordinates: Overview, Categorization and Guidelines for Future Research.IEEE Transactions on Visualization and Computer Graphics, 22(1):579– 588, 2016. doi: 10.1109/TVCG.2015.2466992 2

    2016

  14. [25]

    Kahng and D

    M. Kahng and D. H. P. Chau. How Does Visualization Help Peo- ple Learn Deep Learning? Evaluating GAN Lab with Observational Study and Log Analysis. InIEEE Visualization Conference (VIS), pp. 266–270. IEEE, 2020. doi: 10.1109/VIS47514.2020.00060 2

    2020

  15. [26]

    Y .-a. Kang, C. Gorg, and J. Stasko. Evaluating visual analytics sys- tems for investigative analysis: Deriving design principles from a case study. InIEEE Symposium on Visual Analytics Science and Technol- ogy, pp. 139–146. IEEE, 2009. doi: 10.1109/V AST.2009.5333878 2

    2009

  16. [27]

    Y .-a. Kang, C. Görg, and J. Stasko. How Can Visual Analytics As- sist Investigative Analysis? Design Implications from an Evalua- tion.IEEE Transactions on Visualization and Computer Graphics, 17(5):570–583, 2011. doi: 10.1109/TVCG.2010.84 2

    2011 doi

  17. [28]

    H.-K. Ko, H. Jeon, G. Park, D. H. Kim, N. W. Kim, J. Kim, and J. Seo. Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models, 2024. 2

    2024

  18. [29]

    H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Em- pirical Studies in Information Visualization: Seven Scenarios.IEEE Transactions on Visualization and Computer Graphics, 18(9):1520– 1536, 2012. doi: 10.1109/TVCG.2011.279 2

    2012 doi

  19. [30]

    G. Li, X. Wang, G. Aodeng, S. Zheng, Y . Zhang, C. Ou, S. Wang, and C. H. Liu. Visualization Generation with Large Language Models: An Evaluation. doi: 10.48550/arXiv.2401.11255 2

    doi

  20. [31]

    Z. Li, H. Miao, V . Pascucci, and S. Liu. Visualization Literacy of Multimodal Large Language Models: A Comparative Study.arXiv preprint arXiv:2407.10996, 2024. 1, 2, 8

    2024 arXiv

  21. [32]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved Baselines with Visual Instruction Tuning, 2023. 5

    2023

  22. [33]

    S. Liu, H. Miao, Z. Li, M. Olson, V . Pascucci, and P.-T. Bremer.AVA: Towards Autonomous Visualization Agents through Visual Perception- Driven Decision-Making. 2023. 1, 2

    2023

  23. [35]

    J. Lála, O. O’Donoghue, A. Shtedritski, S. Cox, S. G. Rodriques, and A. D. White. PaperQA: Retrieval-Augmented Generative Agent for Scientific Research, 2023. 2

    2023

  24. [36]

    L. E. Matzen, M. J. Haass, K. M. Divis, Z. Wang, and A. T. Wil- son. Data Visualization Saliency Model: A Tool for Evaluating Ab- stract Data Visualizations.IEEE Transactions on Visualization and Computer Graphics, 24(1):563–573, 2018. doi: 10.1109/TVCG.2017 .2743939 2

    2018 doi

  25. [37]

    GPT-4 Technical Report, 2023

    OpenAI. GPT-4 Technical Report, 2023. 1, 2

    2023

  26. [38]

    Open Science Framework - OSF, 2024

    OSF. Open Science Framework - OSF, 2024. 1

    2024

  27. [39]

    Reimann, C

    D. Reimann, C. Blech, N. Ram, and R. Gaschler. Visual Model Fit Estimation in Scatterplots: Influence of Amount and Decentering of Noise.IEEE Transactions on Visualization and Computer Graphics, 27(9):3834–3838, 2021. doi: 10.1109/TVCG.2021.3051853 3, 4, 6, 7

    2021

  28. [40]

    Sarma, S

    A. Sarma, S. Guo, J. Hoffswell, R. Rossi, F. Du, E. Koh, and M. Kay. Evaluating the Use of Uncertainty Visualisations for Imputations of Data Missing At Random in Scatterplots.IEEE Transactions on Vi- sualization and Computer Graphics, 29(1):602–612, 2023. doi: 10. 1109/TVCG.2...

    2023

  29. [41]

    Z. Shao, L. Shen, H. Li, Y . Shan, H. Qu, Y . Wang, and S. Chen. Nar- rative Player: Reviving Data Narratives with Visuals.IEEE Transac- tions on Visualization and Computer Graphics, pp. 1–15, 2025. doi: 10.1109/TVCG.2025.3530512 2

    2025

  30. [42]

    L. Shen, Y . Zhang, H. Zhang, and Y . Wang. Data Player: Automatic Generation of Data Videos with Narration-Animation Interplay.IEEE Transactions on Visualization and Computer Graphics, 30(1):109– 119, 2024. doi: 10.1109/TVCG.2023.3327197 2

    2024

  31. [43]

    D. A. Szafir, R. Borgo, M. Chen, D. J. Edwards, B. Fisher, and L. Padilla.Visualization Psychology. Springer Nature, 2023. 8

    2023

  32. [44]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models, 2023. 2

    2023

  33. [45]

    Verma, K

    A. Verma, K. Mukherjee, C. Potts, E. Kreiss, and J. E. Fan. Evaluating human and machine understanding of data visualizations. InProceed- ings of the Annual Meeting of the Cognitive Science Society, vol. 46,

  34. [46]

    F. Wang, B. Wang, X. Shu, Z. Liu, Z. Shao, C. Liu, and S. Chen. ChartInsighter: An Approach for Mitigating Hallucination in Time- series Chart Summary Generation with A Benchmark Dataset.IEEE Transactions on Visualization and Computer Graphics, pp. 1–11,

  35. [48]

    H. W. Wang, J. Hoffswell, S. M. T. Thane, V . S. Bursztyn, and C. X. Bearfield. How Aligned are Human Chart Takeaways and LLM Pre- dictions? A Case Study on Bar Charts with Varying Layouts.IEEE Transactions on Visualization and Computer Graphics, 31(1):536– 546, 2025. doi: 10....

    2025

  36. [49]

    L. Wang, S. Zhang, Y . Wang, E.-P. Lim, and Y . Wang. LLM4Vis: Explainable Visualization Recommendation using ChatGPT, 2023. 1, 2

    2023

  37. [50]

    J. Xia, Y . Zhang, J. Song, Y . Chen, Y . Wang, and S. Liu. Revisiting Di- mensionality Reduction Techniques for Visual Cluster Analysis: An Empirical Study.IEEE Transactions on Visualization and Computer Graphics, 28(1):529–539, 2021. doi: 10.1109/TVCG.2021.3114694 2

    2021

  38. [51]

    Xu and E

    Z. Xu and E. Wall. Exploring the Capability of LLMs in Performing Low-Level Visual Analytic Tasks on SVG Data Visualizations.arXiv preprint arXiv:2404.19097, 2024. 2

    2024 arXiv

  39. [52]

    F. Yang, Y . Ma, L. Harrison, J. Tompkin, and D. H. Laidlaw. How Can Deep Neural Networks Aid Visualization Perception Research? Three Studies on Correlation Judgments in Scatterplots. InProceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–17. Associa...

  40. [53]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V,

  41. [54]

    Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang. The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision),

  42. [55]

    A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y . Dong, and J. Tang. AgentTuning: Enabling Generalized Agent Abilities for LLMs, 2023. 2

    2023

  43. [56]

    Y . Zhao, Y . Zhang, Y . Zhang, X. Zhao, J. Wang, Z. Shao, C. Turkay, and S. Chen. LEV A: Using Large Language Models to Enhance Visual Analytics.IEEE Transactions on Visualization and Computer Graph- ics, 31(3):1830–1847, 2025. doi: 10.1109/TVCG.2024.3368060 2

    2025

  44. [57]

    ROLE" refers to the desired background role for the large model,

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su. GPT-4V(ision) is a Generalist Web Agent, if Grounded, 2024. 2 APPENDIX In the first section, we present the prompt and codes for two ex- periments (Fit Estimation and Imputation for Uncertainty) as refer- ences. In the second secti...

    2024

  45. [2023]

    doi: 10.1145/3544548.3581111 1, 2

  46. [2024]

    doi: 10.1109/TVCG.2024.3463800 2

    2024

  47. [2025]

    doi: 10.1109/TVCG.2025.3567122 2

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.