Pith. sign in

REVIEW 2 major objections 6 minor 83 references

DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that open-ended dashboard generation can be judged by replaying each model's self-declared interaction walkthrough, and presents DashArena, a 234-task benchmark, as evidence the protocol works.

desk verdict A serious, well-engineered benchmark for open-ended dashboard generation; the trajectory-evidence protocol is genuinely novel, though the human calibration shares the trajectory with the judge, so one more validation study is needed before the validity claim is fully trusted. read the letter →

arxiv 2608.10567 v1 pith:7MHR6EYM submitted 2026-08-11 cs.AI

Xiaotong Wang , Dazhen Deng This is my paper · ORCID
classification cs.AI
keywords dashboardgenerationinteractivedashboardsLLMevaluationopen-endedbenchmarkreplayableinteractiontrajectoryvision-languagejudgeBradley-TerryaggregationDashJudge-8B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DashArena sets out to measure something earlier benchmarks do not: whether an LLM-generated dashboard genuinely supports the analysis it was asked to support, interactively, not just whether it renders or matches a template. Its answer is to make each model submit both a dashboard and a replayable interaction trajectory—a structured walkthrough of the analytical steps the model intends its dashboard to support. A browser executor replays that trajectory and records which actions succeed and which charts, controls, and texts actually change, turning a model's self-description into checkable evidence. A vision-language judge then compares two anonymous candidates on the same task using this evidence, and Bradley–Terry aggregation turns the pairwise preferences into a leaderboard. Human calibration on 99 pairs shows the open-weight judge DashJudge-8B reproduces human majority preferences at 79.8% agreement ($\kappa=0.600$), while deterministic execution rules reach only 42.4%; the best model still replays only 73.3% of its trajectories, so the paper concludes the task is far from solved.

What carries the argument

The load-bearing mechanism is the replayable interaction trajectory: an ordered JSON list of steps, each specifying an action type, a target identified by a stable data-test-id, a natural-language analytical intent, and an expected visual change. A browser-based executor replays these steps, validates the schema, captures page screenshots, and summarizes which charts, controls, and text components changed after each action; only those screenshots and execution summaries are shown to the judge. Pairwise comparison by a vision-language judge—first a proprietary teacher, then the distilled open-weight DashJudge-8B—converts this evidence into preferences, and Bradley–Terry aggregation converts the preferences into the leaderboard.

What would settle it

Give the same models dashboards containing deliberately planted broken controls in unseen tasks, have them author trajectories, and measure whether broken controls are underrepresented in those trajectories; if the concealment rate is substantially higher than the audit's 90.1%-versus-86.0% authored/unauthored response gap, then trajectory fidelity—and the benchmark's validity—is not assured.

Watch

Extended reading notes

Core claim

The paper's central claim is that interaction quality in open-ended dashboard generation can be evaluated without an exploratory agent: the model itself declares what should be attempted and what change it expects, an independent browser executor verifies what actually happens, and a vision-language judge determines whether the demonstrated behavior serves the task. This replay-before-judging protocol is validated three ways. Interaction evidence improves judge agreement by 8.1 points (from 71.7% to 79.8%) and by 0.159 in Cohen's $\kappa$; the distilled DashJudge-8B matches a proprietary teacher's human alignment while running on open weights; and a 100-candidate audit finds that authored trajectories reach 80.8% per-page control coverage and that authors' chosen controls respond downstream 90.1% of the time versus 86.0% for unauthored controls, giving no evidence of systematic concealment. On the 120-task held-out leaderboard, no model exceeds an 85.8% render rate or a 73.3% replay rate, and manual inspection finds data-binding, analytical, and readability defects even in execution-clean candidates.

Load-bearing premise

The protocol's validity rests on the assumption that a model's self-written interaction walkthrough is a faithful demonstration of what the dashboard can actually do, not a cherry-picked tour that avoids the dashboard's weak points.

Editorial extensions

If this is right

  • Future dashboard-generation benchmarks should require replayable trajectories, because static screenshots miss the 8.1-point drop in judge agreement that interaction evidence provides.
  • Replayability is a distinct bottleneck from rendering: the best model renders 85.8% of dashboards but replays only 73.3% of its trajectories, so interaction failures are not captured by page-load success alone.
  • Because deterministic execution rules reach only 42.4% human agreement, raw action success is not a substitute for semantic judgment; both execution evidence and multimodal reasoning are needed.
  • The leaderboard ordering is stable across Bradley–Terry, Thurstone–Mosteller, and average-win-rate aggregation, so the reported capability tiers do not depend on the choice of aggregation model.
  • Since DashJudge-8B is open-weight and trained on 255 pairs, the evaluation is reproducible on local hardware and can be rerun as new models appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The replayable-trajectory protocol could generalize beyond dashboards to other open-ended interactive artifacts such as web apps, data-analysis notebooks, and design tools, where a model's declared usage plan can be replayed and judged.
  • The paper's own numbers suggest that most remaining judge disagreement lives in genuinely ambiguous pairs: agreement is 92.3% on unanimous pairs but 66.0% on two-to-one majorities, so a follow-up could model preference distributions instead of a single majority label.
  • Because the audit's no-concealment result is a post hoc sample of 100 candidates, a straightforward stress test would be to incentivize models to hide known failures and check whether per-page control coverage remains near 80%.
  • The small 90.1%-versus-86.0% downstream-response gap between authored and unauthored controls implies that omitted controls are almost as functional as demonstrated ones, so a cheaper evaluation might combine execution reports with lightweight heuristics, though the 42.4% deterministic-rule agreement warns against going fully automatic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. DashArena is a benchmark for open-ended, task-grounded generation of interactive analytic dashboards. The authors construct 234 tasks from human-authored Tableau Public dashboards, ask each evaluated model to produce a single-file HTML dashboard plus a structured interaction trajectory, replay the trajectory in a Playwright executor to obtain deterministic execution evidence, and use a VLM judge (distilled into the open-weight DashJudge-8B from a Claude Opus 4.6 teacher) to make pairwise preference judgments aggregated with a Bradley–Terry model. The paper reports a human calibration study (99 evaluable pairs; 79.8% agreement, Cohen's κ=0.600), an ablation showing that interaction evidence improves judge–human agreement by 8.1 points, deterministic-rule comparisons (42.4% agreement), a 100-candidate trajectory-coverage audit, an 841-step failure audit, counterfactual control probes, and leaderboard robustness checks under alternative aggregation rules and task resampling. On 120 held-out tasks, GPT-5.5 ranks first, followed by the human baseline and GLM-5.2, with Grok 4.3 last; the best render and replay rates are 85.8% and 73.3%, respectively.

Significance. The contribution is substantial if the central validity claim holds. DashArena appears to be the first benchmark that evaluates open-ended interactive dashboard generation with reproducible runtime evidence, and the release of DashJudge-8B, tasks, and an anonymized human baseline makes the protocol usable by others. The paper's empirical effort is a strength: the human calibration is larger than typical, the trajectory ablation isolates the effect of interaction evidence, the 100-candidate audit and step-level inspection directly address the possibility of selective trajectory authorship, and the leaderboard is shown to be robust across aggregation rules and task resampling. The deterministic renderable/replayable metrics and the qualitative failure analysis also provide a useful bridge between preference judgments and concrete runtime defects. The main risk is external validity: because the human calibration shares the trajectory evidence with the judge, the reported agreement may reflect agreement about the persuasiveness of a walkthrough rather than about the full analytical utility of the dashboard.

major comments (2)
  1. [Do Model-Authored Trajectories Test Their Dashboards?] The human calibration protocol gives annotators exactly the trajectory and execution report that DashJudge-8B consumes, so the 79.8% agreement and κ=0.600 in Table 2 validate agreement about the submitted walkthrough, not necessarily about the full analytical support of the dashboard. The audit in Table 4 shows that authored trajectories cover 80.8% of controls and that unauthored controls respond comparably when probed, but it does not test whether omitted analytical paths would change a preference; a cherry-picked trajectory can still omit the most demanding comparisons while satisfying the coverage metrics. I request either a free-exploration sub-study in which annotators interact with the dashboards directly, or a sensitivity check in which a stratified subset is re-judged using a second, independently authored trajectory. Without such a check, the load-bearing claim that DashArena measures dashboard utility rather than trajectory persuasiveness remains under-supported.
  2. [Human baseline / Open-weight Judge Distillation] Claude Opus 4.6 is used to author task seeds, to generate the human-baseline trajectory, to teach DashJudge-8B, and as an evaluated candidate. The human baseline is therefore not a matched-protocol human artifact, and its trajectory may encode the same model's preferences used to train the judge. Because the baseline ranks second with a confidence interval overlapping GPT-5.5 (Figure 2a), this is not purely cosmetic. Please either add a truly human-authored baseline trajectory for a subset of tasks or demonstrate that the leaderboard is unchanged when baseline trajectories are produced by a different authoring model.
minor comments (6)
  1. [Human baseline] The sentence 'One author manually verify that every generated trajectory...' contains a typo; it should read 'verifies'. The same paragraph should state explicitly that the human baseline's execution report is simulated rather than produced by replaying the Tableau workbook.
  2. [Open-weight Judge Distillation] Please clarify whether the 20 judge-test tasks are included in the 120-task leaderboard and whether the 45 judge-test pairs are part of the 3,325 aggregated comparisons; the phrase 'rather than an additional split' is ambiguous.
  3. [Table 1] The cell symbols are garbled in the submitted text (e.g., 'Code-only! % % % %'); please render check/cross marks unambiguously and add a legend.
  4. [Interaction Trajectory and Execution] State the maximum trajectory length and the fixed delay per action, since the audit attributes coverage gaps to the single-trajectory length limit.
  5. [Is DashJudge-8B Reliable?] Report the kappa value for the Gemini/DashJudge-8B agreement (currently only the percentage is given), because agreement percentages are sensitive to base rates.
  6. [Introduction] The phrase 'task-clustered Bradley–Terry aggregation' is used in the Introduction, but the experiments describe confidence intervals that resample tasks; clarify whether clustering is used in model fitting or only in resampling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark outputs are measured, not fitted inputs; the remaining trajectory-fidelity concern is a construct-validity caveat, not a circularity.

full rationale

DashArena's load-bearing quantities are not defined in terms of the claims they support. Task seeds are derived from human-authored Tableau dashboards; model candidates generate dashboards and trajectories; the Playwright executor produces deterministic replay evidence; and DashJudge-8B is trained on teacher labels and then validated against held-out human majority preferences (99 pairs, 79.8% agreement, kappa = 0.600) that are disjoint from SFT and leaderboard data. Ablations and deterministic baselines (71.7% and 42.4% agreement) act as independent comparators rather than being folded into the pairwise preferences. The audit of model-authored trajectories probes unauthored controls (90.1% vs. 86.0% downstream response), so trajectory fidelity is tested against external evidence rather than assumed. Citations to prior work by one of the authors (DashBot, ChartGPT) are background only and not load-bearing. The skeptic's point that the human calibration and the judge share trajectory evidence concerns construct validity--whether agreement tracks trajectory persuasiveness rather than underlying dashboard utility--but this is a measurement-interpretation risk, not a case where a result equals its input by construction. No equation or fitted parameter is reused as its own prediction, and no specific reduction to inputs can be exhibited. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on empirical assumptions about judge reliability, trajectory faithfulness, and task-source representativeness rather than on fitted free parameters. No new physical or conceptual entities are introduced. The human calibration study and audits provide partial grounding for the domain assumptions, but all four axioms are load-bearing and none is formally proven.

assumptions (4)
  • domain assumption A VLM judge, given task, screenshots, and execution reports, can assess the relative quality of two dashboards approximately as well as human annotators.
    This is the core assumption behind DashJudge-8B and the leaderboard. The paper validates it on 99 human-labeled pairs (79.8% agreement, kappa 0.600), but the validation set is small and skewed; the assumption remains an empirical claim, not a proven property.
  • domain assumption A generator-authored interaction trajectory, when replayed by the Playwright executor, is a faithful and non-adversarial demonstration of the artifact's intended analytical workflow.
    Motivates the whole design (section 'Why model-authored trajectories?'). The paper's own audits show 80.8% per-page control coverage and 6.7% step-level errors, so trajectories are informative but not exhaustive; the assumption that omitted controls are not concealed failures is tested but not guaranteed.
  • domain assumption The Tableau Public dashboards used as task seeds are representative of realistic analytic dashboards.
    The authors acknowledge platform-specific bias in the Limitations: 'Tableau Public carries platform-specific biases, leaving private and other visualization ecosystems outside the current scope.'
  • standard math Bradley-Terry model with independent candidate-order randomization yields a valid latent-ability ranking for paired comparison data.
    Eq. (1) defines P(i dominates j) = sigma(theta_i minus theta_j); the paper checks robustness against Thurstone-Mosteller and average win rate, finding the same ordering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation." pith.science (2026). https://pith.science/paper/7MHR6EYM

@misc{pith2026260810567,
  author       = {Pith},
  title        = {Pith review of: DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MHR6EYM}},
  note         = {Machine review of arXiv:2608.10567}
}
read the original abstract

Analytic dashboards combine coordinated views and interactions for data exploration and decision-making. Recent models can generate them from data and natural-language goals, but evaluating their usefulness remains difficult. Dashboard generation is open-ended, and neither static appearance nor successful execution alone captures analytical support and interaction quality. We introduce DashArena, to our knowledge the first benchmark for open-ended, task-grounded generation of interactive analytic dashboards. Its key innovation is to require each system to generate both a dashboard and a replayable interaction trajectory. A browser executor replays the trajectory and turns the system's intended analytical workflow into reproducible visual and execution evidence. A VLM judge compares candidates using this evidence, and Bradley--Terry aggregation produces the leaderboard. We further distill the judge into the open-weight DashJudge-8B. Human evaluations show that DashJudge-8B effectively reproduces human judgments and ablations show that interaction evidence improves judge agreement. Experiments with frontier models reveal persistent rendering, analytical, and interaction failures. Together, these results show that realistic dashboard generation remains challenging and that interaction-aware evaluation captures failures missed by static or execution-only checks.

Figures

Figures reproduced from arXiv: 2608.10567 by the authors.

Figure 1
Figure 1. DashArena overview. Human-authored dashboards [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Main leaderboard on 120 held-out tasks from 3,325 directional DashJudge-8B preferences. (a) Scaled Bradley–Terry [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 48 canonical work pages

  1. [1]

    IEEE Transactions on Visualization and Computer Graphics , volume=

    Dashboard Design Patterns , author=. IEEE Transactions on Visualization and Computer Graphics , volume=. 2023 , publisher=

    2023

  2. [2]

    IEEE Transactions on Visualization and Computer Graphics , volume=

    What Do We Talk About When We Talk About Dashboards? , author=. IEEE Transactions on Visualization and Computer Graphics , volume=. 2019 , publisher=

    2019

  3. [3]

    IEEE Transactions on Visualization & Computer Graphics , volume=

    Heuristics for supporting cooperative dashboard design , author=. IEEE Transactions on Visualization & Computer Graphics , volume=. 2024 , doi=

    2024

  4. [4]

    The Craft of Information Visualization , pages=

    The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations , author =. The Craft of Information Visualization , pages=. 2003 , publisher =

    work page 2003

  5. [5]

    Fifth International Conference on Coordinated and Multiple Views in Exploratory Visualization (CMV 2007) , pages=

    State of the Art: Coordinated & Multiple Views in Exploratory Visualization , author=. Fifth International Conference on Coordinated and Multiple Views in Exploratory Visualization (CMV 2007) , pages=. 2007 , volume=

    work page 2007

  6. [6]

    IEEE Transactions on Visualization and Computer Graphics , volume=

    Composition and Configuration Patterns in Multiple-View Visualizations , author=. IEEE Transactions on Visualization and Computer Graphics , volume=. 2021 , doi=

    work page 2021

  7. [7]

    Cutter IT journal , volume=

    Effective dashboard design , author=. Cutter IT journal , volume=

    work page

  8. [8]

    IEEE Transactions on Visualization and Computer Graphics , volume=

    Nebula: A Coordinating Grammar of Graphics , author=. IEEE Transactions on Visualization and Computer Graphics , volume=. 2022 , doi=

    work page 2022

Show all 83 references
  1. [9]

    2006 , isbn =

    Few, Stephen , title =. 2006 , isbn =

    2006

  2. [10]

    Proceedings of the 13th IFIP TC 13 International Conference on Human-Computer Interaction - Volume Part IV , pages =

    Elias, Micheline and Bezerianos, Anastasia , title =. Proceedings of the 13th IFIP TC 13 International Conference on Human-Computer Interaction - Volume Part IV , pages =. 2011 , isbn =

    2011

  3. [11]

    IEEE Transactions on Visualization and Computer Graphics , volume=

    From Dashboard Zoo to Census: A Case Study With Tableau Public , author=. IEEE Transactions on Visualization and Computer Graphics , volume=. 2025 , doi=

    2025

  4. [12]

    2025 , isbn =

    Lisnic, Maxim and Setlur, Vidya and Sultanum, Nicole , title =. 2025 , isbn =. doi:10.1145/3706598.3713580 , booktitle =

    2025

  5. [13]

    Mosaic: An Architecture for Scalable & Interoperable Data Views , year=

    Heer, Jeffrey and Moritz, Dominik , journal=. Mosaic: An Architecture for Scalable & Interoperable Data Views , year=

  6. [14]

    2025 , issue_date =

    Tian, Yuan and Cui, Weiwei and Deng, Dazhen and Yi, Xinjing and Yang, Yurun and Zhang, Haidong and Wu, Yingcai , title =. 2025 , issue_date =. doi:10.1109/TVCG.2024.3368621 , journal =

    2025

  7. [15]

    DMiner: Dashboard Design Mining and Recommendation , year=

    Lin, Yanna and Li, Haotian and Wu, Aoyu and Wang, Yong and Qu, Huamin , journal=. DMiner: Dashboard Design Mining and Recommendation , year=

  8. [16]

    IEEE Transactions on Visualization and Computer Graphics , month = jul, pages =

    Feng, Yingchaojie and Wang, Xingbo and Pan, Bo and Wong, Kam Kwai and Ren, Yi and Liu, Shi and Yan, Zihan and Ma, Yuxin and Qu, Huamin and Chen, Wei , title =. IEEE Transactions on Visualization and Computer Graphics , month = jul, pages =. 2024 , issue_date =. doi:10.1109/TVC...

    2024

  9. [17]

    and Manz, Trevor and Gehlenborg, Nils , booktitle=

    Keller, Mark S. and Manz, Trevor and Gehlenborg, Nils , booktitle=. Use-Coordination: Model, Grammar, and Library for Implementation of Coordinated Multiple Views , year=

  10. [18]

    An Empirical Guide for Visualization Consistency in Multiple Coordinated Views , year=

    Tan, Shaocong and Lai, Chufan and Zhang, Xiaolong Luke and Yuan, Xiaoru , booktitle=. An Empirical Guide for Visualization Consistency in Multiple Coordinated Views , year=

  11. [19]

    Keeping Multiple Views Consistent: Constraints, Validations, and Exceptions in Visualization Authoring , year=

    Qu, Zening and Hullman, Jessica , journal=. Keeping Multiple Views Consistent: Constraints, Validations, and Exceptions in Visualization Authoring , year=

  12. [20]

    and Lin, Halden and Smith, Adam M

    Moritz, Dominik and Wang, Chenglong and Nelson, Greg L. and Lin, Halden and Smith, Adam M. and Howe, Bill and Heer, Jeffrey , journal=. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco , year=

  13. [21]

    2023 IEEE Visualization and Visual Analytics (VIS) , title=

    Yang, Junran and Gyarmati, P. 2023 IEEE Visualization and Visual Analytics (VIS) , title=. 2023 , volume=

    2023

  14. [22]

    2025 , issue_date =

    Wang, Huichen Will and Gordon, Mitchell and Battle, Leilani and Heer, Jeffrey , title =. 2025 , issue_date =. doi:10.1109/TVCG.2024.3456350 , journal =

    2025

  15. [23]

    LightVA: Lightweight Visual Analytics With LLM Agent-Based Task Planning and Execution , year=

    Zhao, Yuheng and Wang, Junjie and Xiang, Linbing and Zhang, Xiaowen and Guo, Zifei and Turkay, Cagatay and Zhang, Yu and Chen, Siming , journal=. LightVA: Lightweight Visual Analytics With LLM Agent-Based Task Planning and Execution , year=

  16. [24]

    DataShot: Automatic Generation of Fact Sheets from Tabular Data , year=

    Wang, Yun and Sun, Zhida and Zhang, Haidong and Cui, Weiwei and Xu, Ke and Ma, Xiaojuan and Zhang, Dongmei , journal=. DataShot: Automatic Generation of Fact Sheets from Tabular Data , year=

  17. [25]

    arXiv preprint arXiv:2504.12865 , year=

    DashChat: Interactive Authoring of Industrial Dashboard Design Prototypes through Conversation with LLM-Powered Agents , author=. arXiv preprint arXiv:2504.12865 , year=

    arXiv

  18. [26]

    SmartMLVs: LLM-enabled Multiple Linked Views Generation for Interactive Visualization , year=

    Qiu, Tian and Wang, Fen and Huang, Shaohua and Guo, Meng and Zhao, Yuheng and Li, Jincheng and Chen, Siming , booktitle=. SmartMLVs: LLM-enabled Multiple Linked Views Generation for Interactive Visualization , year=

  19. [27]

    arXiv preprint arXiv:2601.06126 , year=

    NL2Dashboard: A Lightweight and Controllable Framework for Generating Dashboards with LLMs , author=. arXiv preprint arXiv:2601.06126 , year=

  20. [28]

    The Eleventh International Conference on Learning Representations , year=

    React: Synergizing reasoning and acting in language models , author=. The Eleventh International Conference on Learning Representations , year=

  21. [29]

    Vega-Lite: A Grammar of Interactive Graphics , year=

    Satyanarayan, Arvind and Moritz, Dominik and Wongsuphasawat, Kanit and Heer, Jeffrey , journal=. Vega-Lite: A Grammar of Interactive Graphics , year=

  22. [30]

    MultiVision: Designing Analytical Dashboards with Deep Learning Based Recommendation , year=

    Wu, Aoyu and Wang, Yun and Zhou, Mengyu and He, Xinyi and Zhang, Haidong and Qu, Huamin and Zhang, Dongmei , journal=. MultiVision: Designing Analytical Dashboards with Deep Learning Based Recommendation , year=

  23. [31]

    DashBot: Insight-Driven Dashboard Generation Based on Deep Reinforcement Learning , year=

    Deng, Dazhen and Wu, Aoyu and Qu, Huamin and Wu, Yingcai , journal=. DashBot: Insight-Driven Dashboard Generation Based on Deep Reinforcement Learning , year=

  24. [32]

    MEDLEY: Intent-based Recommendations to Support Dashboard Composition , year=

    Pandey, Aditeya and Srinivasan, Arjun and Setlur, Vidya , journal=. MEDLEY: Intent-based Recommendations to Support Dashboard Composition , year=

  25. [33]

    and Cleveland, William S

    Becker, Richard A. and Cleveland, William S. , title =. 1987 , issue_date =. doi:10.2307/1269768 , journal =

    1987 doi

  26. [34]

    Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations , year=

    Wongsuphasawat, Kanit and Moritz, Dominik and Anand, Anushka and Mackinlay, Jock and Howe, Bill and Heer, Jeffrey , journal=. Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations , year=

  27. [35]

    2017 , isbn =

    Wongsuphasawat, Kanit and Qu, Zening and Moritz, Dominik and Chang, Riley and Ouk, Felix and Anand, Anushka and Mackinlay, Jock and Howe, Bill and Heer, Jeffrey , title =. 2017 , isbn =. doi:10.1145/3025453.3025768 , booktitle =

    2017

  28. [36]

    1986 , issue_date =

    Mackinlay, Jock , title =. 1986 , issue_date =. doi:10.1145/22949.22950 , journal =

    1986

  29. [37]

    Data2Vis: Automatic Generation of Data Visualizations Using Sequence-to-Sequence Recurrent Neural Networks , year=

    Dibia, Victor and Demiralp, Çağatay , journal=. Data2Vis: Automatic Generation of Data Visualizations Using Sequence-to-Sequence Recurrent Neural Networks , year=

  30. [38]

    and Li, Stephen and Kraska, Tim and Hidalgo, C\'

    Hu, Kevin and Bakker, Michiel A. and Li, Stephen and Kraska, Tim and Hidalgo, C\'. VizML: A Machine Learning Approach to Visualization Recommendation , year =. doi:10.1145/3290605.3300358 , booktitle =

  31. [39]

    NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries , year=

    Narechania, Arpit and Srinivasan, Arjun and Stasko, John , journal=. NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries , year=

  32. [40]

    2007 , issue_date =

    Mackinlay, Jock and Hanrahan, Pat and Stolte, Chris , title =. 2007 , issue_date =. doi:10.1109/TVCG.2007.70594 , journal =

    2007

  33. [41]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=

    LIDA: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=

  34. [42]

    2025 , address=

    Ouyang, Geliang and Chen, Jingyao and Nie, Zhihe and Gui, Yi and Wan, Yao and Zhang, Hongyu and Chen, Dongping , booktitle=. 2025 , address=. doi:10.18653/v1/2025.acl-long.960 , url=

    2025 doi

  35. [43]

    2000 , isbn =

    North, Chris and Shneiderman, Ben , title =. 2000 , isbn =. doi:10.1145/345513.345282 , booktitle =

    2000

  36. [44]

    arXiv preprint arXiv:2112.12926 , year=

    nvBench: A large-scale synthesized dataset for cross-domain natural language to visualization task , author=. arXiv preprint arXiv:2112.12926 , year=

    arXiv

  37. [45]

    2025 , url=

    Luo, Tianqi and Huang, Chuhan and Shen, Leixian and Li, Boyan and Shen, Shuyu and Zeng, Wei and Tang, Nan and Luo, Yuyu , booktitle=. 2025 , url=

    2025

  38. [46]

    2021 , isbn =

    Zhou, Mengyu and Li, Qingtao and He, Xinyi and Li, Yuejiang and Liu, Yibo and Ji, Wei and Han, Shi and Chen, Yining and Jiang, Daxin and Zhang, Dongmei , title =. 2021 , isbn =. doi:10.1145/3447548.3467279 , booktitle =

    2021

  39. [47]

    DeepEye: Towards Automatic Data Visualization , year=

    Luo, Yuyu and Qin, Xuedi and Tang, Nan and Li, Guoliang , booktitle=. DeepEye: Towards Automatic Data Visualization , year=

  40. [48]

    VisEval: A Benchmark for Data Visualization in the Era of Large Language Models , year=

    Chen, Nan and Zhang, Yuge and Xu, Jiahang and Ren, Kan and Yang, Yuqing , journal=. VisEval: A Benchmark for Data Visualization in the Era of Large Language Models , year=

  41. [49]

    International Conference on Learning Representations , year=

    VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations , author=. International Conference on Learning Representations , year=

  42. [50]

    Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models , year=

    Maddigan, Paula and Susnjak, Teo , journal=. Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models , year=

  43. [51]

    1999 , isbn =

    Horvitz, Eric , title =. 1999 , isbn =. doi:10.1145/302979.303030 , booktitle =

    1999

  44. [52]

    and Guinn, C.I

    Allen, J.E. and Guinn, C.I. and Horvtz, E. , journal=. Mixed-initiative interaction , year=

  45. [53]

    and Tory, Melanie and Gossweiler, Rich and Chang, Angel X

    Setlur, Vidya and Battersby, Sarah E. and Tory, Melanie and Gossweiler, Rich and Chang, Angel X. , title =. 2016 , isbn =. doi:10.1145/2984511.2984588 , booktitle =

    2016

  46. [54]

    , title =

    Gao, Tong and Dontcheva, Mira and Adar, Eytan and Liu, Zhicheng and Karahalios, Karrie G. , title =. 2015 , isbn =. doi:10.1145/2807442.2807478 , booktitle =

    2015

  47. [55]

    Proceedings of the 10th International Conference on Smart Graphics , pages =

    Sun, Yiwen and Leigh, Jason and Johnson, Andrew and Lee, Sangyoon , title =. Proceedings of the 10th International Conference on Smart Graphics , pages =. 2010 , isbn =

    2010

  48. [56]

    1981 , isbn =

    Shneiderman, Ben , title =. 1981 , isbn =. doi:10.1145/800276.810991 , booktitle =

    1981

  49. [57]

    SIGSOC Bull

    Shneiderman, Ben , title =. SIGSOC Bull. , month = may, pages =. 1981 , issue_date =. doi:10.1145/1015579.810991 , abstract =

    1981

  50. [58]

    Proceedings of the 16th Eurographics Conference on Visualization , pages =

    Satyanarayan, Arvind and Heer, Jeffrey , title =. Proceedings of the 16th Eurographics Conference on Visualization , pages =. 2014 , publisher =

    2014

  51. [59]

    Lyra 2: Designing Interactive Visualizations by Demonstration , year=

    Zong, Jonathan and Barnwal, Dhiraj and Neogy, Rupayan and Satyanarayan, Arvind , journal=. Lyra 2: Designing Interactive Visualizations by Demonstration , year=

  52. [60]

    , booktitle=

    Weaver, C. , booktitle=. Metavisual Exploration and Analysis of DEVise Coordination in Improvise , year=

  53. [61]

    2018 , isbn =

    Liu, Zhicheng and Thompson, John and Wilson, Alan and Dontcheva, Mira and Delorey, James and Grigg, Sam and Kerr, Bernard and Stasko, John , title =. 2018 , isbn =. doi:10.1145/3173574.3173697 , booktitle =

    2018

  54. [62]

    and Wang, Sitong and Sahni, Riya and Chilton, Lydia B , title =

    Ma, Jenny GuangZhen and Sreedhar, Karthik and Liu, Vivian and Perez, Pedro A. and Wang, Sitong and Sahni, Riya and Chilton, Lydia B , title =. 2025 , isbn =. doi:10.1145/3706598.3714115 , booktitle =

    2025

  55. [63]

    Brooke, John , title =. J. Usability Studies , month = feb, pages =. 2013 , issue_date =

    2013

  56. [64]

    Advances in psychology , volume=

    Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research , author=. Advances in psychology , volume=. 1988 , publisher=

    1988

  57. [65]

    W3C recommendation , volume=

    Document object model (dom) level 1 specification , author=. W3C recommendation , volume=

  58. [66]

    2018 , publisher=

    Content analysis: An introduction to its methodology , author=. 2018 , publisher=

    2018

  59. [67]

    and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle=

    Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle=. 2024 , url=

    2024

  60. [68]

    arXiv preprint arXiv:2401.05507 , year=

    Infiagent-dabench: Evaluating agents on data analysis tasks , author=. arXiv preprint arXiv:2401.05507 , year=

    arXiv

  61. [69]

    arXiv preprint arXiv:2606.17861 , year=

    GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? , author=. arXiv preprint arXiv:2606.17861 , year=

    arXiv

  62. [70]

    Proceedings of the ACM Conference on AI and Agentic Systems , pages=

    Vibe Code Bench: Evaluating AI models on end-to-end web application development , author=. Proceedings of the ACM Conference on AI and Agentic Systems , pages=

  63. [71]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Gui agents: A survey , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  64. [72]

    the method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. the method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    1952

  65. [73]

    arXiv preprint arXiv:2506.13326 , year=

    VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation , author=. arXiv preprint arXiv:2506.13326 , year=

    arXiv

  66. [74]

    2025 , address=

    Si, Chenglei and Zhang, Yanzhe and Li, Ryan and Yang, Zhengyuan and Liu, Ruibo and Yang, Diyi , booktitle=. 2025 , address=. doi:10.18653/v1/2025.naacl-long.199 , url=

    2025 doi

  67. [75]

    2024 , address=

    Koh, Jing Yu and Lo, Robert and Jang, Lawrence and Duvvur, Vikram and Lim, Ming and Huang, Po-Yu and Neubig, Graham and Zhou, Shuyan and Salakhutdinov, Russ and Fried, Daniel , booktitle=. 2024 , address=. doi:10.18653/v1/2024.acl-long.50 , url=

    2024 doi

  68. [76]

    2024 , volume=

    Chen, Dongping and Chen, Ruoxi and Zhang, Shilin and Wang, Yaochen and Liu, Yinuo and Zhou, Huichi and Zhang, Qihui and Wan, Yao and Zhou, Pan and Sun, Lichao , booktitle=. 2024 , volume=

    2024

  69. [77]

    arXiv preprint arXiv:2603.09652 , year=

    MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants , author=. arXiv preprint arXiv:2603.09652 , year=

    arXiv

  70. [78]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  71. [79]

    arXiv preprint arXiv:2507.04952 , year=

    Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation , author=. arXiv preprint arXiv:2507.04952 , year=

  72. [80]

    Scaling , pages=

    A law of comparative judgment , author=. Scaling , pages=. 2017 , publisher=

    2017

  73. [81]

    The least squares solution assuming equal standard deviations and equal correlations , author=

    Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations , author=. Psychometrika , volume=. 1951 , publisher=

    1951

  74. [82]

    1983 , publisher=

    The visual display of quantitative information , author=. 1983 , publisher=

    1983

  75. [83]

    Proceedings of the 34th International Conference on Machine Learning , series=

    On Calibration of Modern Neural Networks , author=. Proceedings of the 34th International Conference on Machine Learning , series=

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.