Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Pith's one-line read LLM planetarium pilots can cut human workload as co-pilots, but cannot replace pilots.

desk verdict An honest little pilot study with a useful error taxonomy, but the headline workload-reduction claim rests on an extrapolation, not on the co-pilot condition the paper actually proposes. read the letter →

arxiv 2601.20466 v2 pith:NEZPI6OF submitted 2026-01-28 cs.HC

classification cs.HC
keywords LLMVisualizationPublicSpacesConversationalAIProactivePlanetariumHuman-AIcollaborationOpenSpace
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that an LLM-based conversational agent can listen to a planetarium guide's speech and execute visualization commands—moving the camera, changing simulation time, and toggling assets—but that this AI pilot lacks the timing, camera control, and audience-reading skills of a human pilot. The central claim is that instead of replacing the human pilot, the AI is best used as a co-pilot: it can handle asset preparation and toggling while the human keeps control of pacing and camera motion, reducing workload. A comparative study with five guides testing the system and two experts providing the human-pilot baseline found mixed preferences between reactive and proactive modes, with proactive feeling more fluid but less reliable. The paper concludes that AI pilots are not a replacement and likely never will be, but are useful for multitasking, preparation, and onboarding novice pilots.

What carries the argument

The carrying mechanism is the conversational AI pilot built on the astrophysics visualization software OpenSpace and a low-latency multimodal language model. It receives the guide's speech (in reactive mode through a hand-triggered microphone, in proactive mode through streaming speech-to-text), interprets the speech as commands, and dispatches tool calls through OpenSpace's Lua API to travel between scene nodes, toggle asset visibility, change simulation time, or do nothing via a 'no-op' call. The comparison between reactive and proactive modes is the experimental pivot.

What would settle it

Run real public planetarium shows with the same AI co-pilot and measure human pilot workload and experimenter interventions; if the AI requires intervention nearly as often as it saves actions, or if guides report no workload drop, the central claim fails. A sharper version would compare AI co-pilot shows head-to-head with human-human pilot pairs on the same measures.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM-based pilot, which listens to a guide's speech and executes commands like moving the camera, changing simulation time, and toggling assets, is not able to replace a human pilot in a live planetarium show. The decisive observation is that the AI is good at toggling assets and preparing actions but poor at camera control and pacing, which demand temporal awareness and precision. The paper therefore proposes a division of labor: keep a human pilot for camera and pacing, and let the AI act as a co-pilot to reduce cognitive load and enable multitasking. Five guides tested reactive and proactive modes; proactive felt more natural but less reliable, while re

Load-bearing premise

The conclusion that AI co-pilots reduce workload rests on the assumption that the experience of five guides in simulated shows—where the experimenter had to intervene repeatedly because the software was unstable—and the expectations of two interviewed experts predict what would happen in real public shows.

Editorial extensions

If this is right

  • Planetarium shows would keep a human pilot for camera control and pacing, while the AI co-pilot handles toggling and preparing visual assets.
  • Proactive listening, once reliability improves, can reduce mental load and preserve narrative flow because guides no longer need to press a trigger.
  • The error dimensions identified—detection, reasoning, context, and naturalness—offer a concrete checklist for building and evaluating AI pilots.
  • AI could assist during show preparation and onboarding, letting novice pilots assemble and test storytelling blocks in a low-risk setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the co-pilot division of labor holds, the same pattern could transfer to other live visualization settings, such as surgery or forensic reconstruction, where a human controls continuous motion while AI preloads context.
  • The paper's 'interaction recommender system' idea suggests a less autonomous intermediate step: the AI suggests actions rather than executing them, which may be more robust before streaming speech and latency improve.
  • The error dimension spectrum could be formalized into a benchmark with human-coded ground truth, making future AI pilot comparisons quantitative rather than interview-based.
  • A reading of the mixed proactive/reactive preference: reliability is the gating factor; if proactive mode's reliability approaches reactive mode's, its fluidity advantage may make it the default.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes the design and evaluation of an LLM-based conversational pilot for the OpenSpace planetarium visualization system, intended to assist guides during live dome shows. The system operates in two modes: a reactive mode in which the guide triggers voice commands with a handheld microphone, and a proactive mode in which the system continuously listens and executes implicit commands. Five professional guides from the authors' institution conducted simulated shortened shows in a real dome, and two additional experts provided interview-based descriptions of human piloting practice. The paper reports qualitative thematic analysis and an error-case taxonomy. The central claim is that, while AI pilots are not yet able to replace human pilots, they could serve as co-pilots to reduce human pilot workload and enable multitasking.

Significance. If the central claim were empirically supported, the work would be a useful early contribution to an emerging area: LLM-driven interaction with visualization in high-stakes, time-critical, public-facing settings. The paper's strengths are its concrete system implementation, the comparison of reactive vs. proactive conversational modes, the honest reporting of failures and mixed participant preferences, and the articulation of an error taxonomy with concrete failure examples. The paper also identifies a genuinely under-explored design direction, interaction recommender systems, that could inform future work. However, the significance is currently limited because the headline conclusion about workload reduction is an extrapolation from subjective impressions and speculative participant statements rather than a measured outcome of the study's design.

major comments (3)
  1. [§5.3 and overall design (RQ1)] The central claim that AI can act as a co-pilot to reduce human workload and enable multitasking is not directly tested. The study compares two AI-only conditions (reactive and proactive), in which the guide is the sole pilot and the AI executes commands; there is no condition in which a human pilot works alongside the AI. The workload-reduction conclusion rests on participants' speculative statements (e.g., P3's satellite anecdote in §5.3) and on a human-pilot baseline reconstructed from two expert interviews, not on any measured comparison. RQ1 is phrased as 'AI co-pilots compared to human co-pilots,' but the experiment never instantiates the co-pilot configuration. To support the claim, the paper needs either a condition with human+AI teaming, or a substantial reframing of the conclusion as a hypothesis for future work rather than a study result.
  2. [§4.2 and §4.3] The evidence base for comparing AI piloting with human piloting is fragile. The human-pilot baseline is based on interviews with two consulting experts (C1, C2) rather than on logged or observed human pilot performance. In the AI conditions, the first author had to intervene several times because of software instability, and P4 could not run the system at all, instead watching P5's session. The paper states in §4.3 that 'at this stage, we focus on reporting on the insights from the interviews' and defers analysis of the collected logs. Consequently, the reported comparisons are qualitative impressions from a small, partly interrupted sample, and the abstract's 'results show' phrasing overstates the evidentiary strength. The paper should clearly label the workload-reduction and co-pilot statements as preliminary hypotheses and temper the abstract accordingly.
  3. [§5.4 and §4.3] The error-case analysis is presented as a substantive result, but it is anecdotal. Section 4.3 says the system logs contain latency, number, and success rate of interventions, yet §4.3 also says the paper focuses on interview insights, and §5.4 explicitly states that a 'qualitative analysis of error cases (both their characteristics, and prevalence) is needed.' No counts, success rates, or inter-rater coding are reported. The four error dimensions are plausible and useful as a taxonomy, but they are not derived from a systematic coding of the logs. Since the paper claims an 'error-case analysis from the system operation logs' in §4, this inconsistency should be resolved—either by reporting quantitative error data or by explicitly labeling the taxonomy as a set of observed examples rather than an analysis.
minor comments (5)
  1. [Abstract and §4.1] The abstract says '7 professional guides,' but §4.1 describes five expert participants (P1–P5) and two consulting experts (C1, C2). The two experts did not use the system. Please clarify the wording to distinguish study participants from expert interviewees.
  2. [§4.2] P4 could not run the system and instead observed P5; the subsequent joint interview means the effective number of independent system-use sessions is four, not five. This should be stated explicitly in the participants and protocol description.
  3. [§5.4] The sentence 'A qualitative analysis of error cases (both their characteristics, and prevalence) is needed' is internally inconsistent: prevalence is a quantitative measure. Moreover, 'is needed' suggests the analysis has not been performed, which conflicts with the earlier claim of an 'error-case analysis from the system operation logs' in §4. Please revise to avoid this contradiction.
  4. [Title page and references] The ACM reference format section contains placeholder dates ('February 2018', 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009') and the author list has inconsistent spacing in the second author's name ('M UJTABA FADHIL JA W AD'). These formatting issues should be corrected.
  5. [§5.2] The finding that the reactive mode 'added to the mental load' is reported without a direct comparison of measured cognitive load. Since cognitive load is a central concept in the paper's argument, it would help to define how the authors infer cognitive load from the interviews and to acknowledge that this is a perceived, not measured, construct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's conclusions are qualitative inductions from interviews, not results forced by definitions, fits, or self-citations.

full rationale

The paper makes no formal derivation or quantitative prediction; its central claim that AI pilots 'could become useful as co-pilots to reduce workload of human pilots and allow multitasking' is an inductive synthesis of participant interviews (Section 5.3) and expert comparisons (Section 5.1), not a quantity computed from the system. No parameter is fitted and then reported as a prediction; no 'uniqueness theorem' or prior claim by the same authors is used to force the conclusion. The self-citations present (e.g., [6] for the system's inspiration, [7] for the state of the art, [18] for prior proactive-agent work) position the work within a lineage but do not carry the conclusion. The manuscript itself flags limitations that reduce evidentiary strength but not circularity: Section 4.2 states 'Due to some instability with the software, the first author had to intervene several times during the presentation' and P4's run failed, and Section 4.3 says 'At this stage, we focus on reporting on the insights from the interviews.' These are reliability/generalizability threats, not cases where the outcome equals an input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study's claims rest on small-sample qualitative assumptions; no fitted parameters or new theoretical entities are introduced.

assumptions (3)
  • domain assumption Professional guides' qualitative feedback is representative of the broader population of planetarium guides.
    All five test participants work at the authors' institution and were offered co-authorship, so their views may be unrepresentative. Invoked throughout Section 5.
  • domain assumption The simulated shortened dome show is a valid proxy for a real public show.
    Sessions were simulated, and software instability led the first author to intervene; real audience dynamics (laughs, gasps, questions) were absent. Section 4.2.
  • domain assumption Human piloting baseline can be captured from two consulting experts' interviews rather than direct measurement.
    No human-pilot condition was run; comparisons to human pilots rely on expert retrospective accounts. Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers." pith.science (2026). https://pith.science/paper/NEZPI6OF

@misc{pith2026260120466,
  author       = {Pith},
  title        = {Pith review of: Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NEZPI6OF}},
  note         = {Machine review of arXiv:2601.20466}
}
read the original abstract

We designed and evaluated an AI pilot in a planetarium visualization software, OpenSpace, for public shows in science centers. The piloting role is usually given to a human working in close collaboration with the guide on stage. We recruited 7 professional guides with extensive experience in giving shows to the public to study the impact of the AI-piloting on the overall experience. The AI-pilot is a conversational AI-agent listening to the guide and interpreting the verbal statements as commands to execute camera motions, change simulation time, or toggle visual assets. Our results show that, while AI pilots lack several critical skills for live shows, they could become useful as co-pilots to reduce workload of human pilots and allow multitasking. We propose research directions toward implementing visualization pilots and co-pilots in live settings.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 2 linked inside Pith

  1. [1]

    Basole and Timothy Major

    Rahul C. Basole and Timothy Major. 2024. Generative AI for visualization: Opportunities and challenges.IEEE Computer Graphics and Applications 44, 2 (2024), 55–64. doi:10/nm2j Manuscript submitted to ACM Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers 7

  2. [2]

    Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Marc Soubeyrand, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic

    Lonni Besançon, Amir Semmo, David J. Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Marc Soubeyrand, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic. 2020. Reducing affective responses to surgical images and videos through stylization.Computer Graphics Forum39, 1 (2020), 462–483. doi:10/gjbm72

  3. [3]

    Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic

    Lonni Besançon, Amir Semmo, David J. Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic

  4. [4]

    Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D

    Marcel Binz, Stephan Alaniz, Adina Roskies, Balazs Aczel, Carl T. Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D. West, Qiong Zhang, Richard M. Shiffrin, Samuel J. Gershman, Vencislav Popov, Emily M. Bender, Marco Marelli, Matthew M. Botvinick, Zeynep Akata, and Eric Schulz

  5. [5]

    Alexander Bock, Emil Axelsson, Jonathas Costa, Gene Payne, Micah Acinapura, Vivian Trakinski, Carter Emmart, Claudio Silva, Charles Hansen, and Anders Ynnerman. 2020. OpenSpace: A system for astrographics.IEEE Transactions on Visualization and Computer Graphics26, 1 (2020), 633–642. doi:10/ggsrbm

  6. [6]

    Mathis Brossier, Alexander Bock, Konrad Schönborn, Tobias Isenberg, Anders Ynnerman, and Lonni Besançon. 2024. In space, no one (but AI) can hear you scream. InPosters at IEEE VIS. doi:10/hbk46z

  7. [7]

    Mathis Brossier, Tobias Isenberg, Konrad Schönborn, Jonas Unger, Mario Romero, Johanna Björklund, Anders Ynnerman, and Lonni Besançon. 2026. State of the Art of LLM-Enabled Interaction with Visualization. arXiv:2601.14943 doi:10/hbk462

  8. [8]

    Mathis Brossier, Robin Skånberg, Lonni Besançon, Mathieu Linares, Tobias Isenberg, Anders Ynnerman, and Alexander Bock. 2023. Moliverse: Contextually embedding the microcosm into the universe.Computers & Graphics112 (2023), 22–30. doi:10/gr7bbz

Show all 36 references
  1. [9]

    Shubham Chatterjee, Xi Wang, Shuo Zhang, Sajad Ebrahimi, Zhaochun Ren, Debasis Ganguly, Gareth Jones, Emine Yilmaz, and Hamed Zamani

  2. [10]

    Juntong Chen, Jiang Wu, Jiajing Guo, Vikram Mohanty, Xueming Li, Jorge Piazentin Ono, Wenbin He, Liu Ren, and Dongyu Liu. 2025. InterChat: Enhancing generative visual analytics using multimodal interactions.Computer Graphics Forum44, 3, Article e70112 (2025), 24 pages. doi:10/g9qr4q

  3. [11]

    Zina Chkirbene, Ridha Hamila, Ala Gouissem, and Unal Devrim. 2024. Large language models (LLM) in industry: A survey of applications, challenges, and trends. InProceedings of the International Conference on Smart Communities: Improving Quality of Life using AI, Robotics and Io...

  4. [12]

    InProceedings of the International Conference on Information and Knowledge Management

    ProActLLM: Proactive conversational information seeking with large language models. InProceedings of the International Conference on Information and Knowledge Management. ACM, New York, 6894–6897. doi:10/qn2b

  5. [13]

    2025.What Does the Public Think About AI? An Overview of the Public’s Attitudes Towards AI and a Resource for Future Research

    Noemi Dreksler, Harry Law, Chloe Ahn, Daniel Schiff, Kaylyn Jackson Schiff, and Zachary Peskowitz. 2025.What Does the Public Think About AI? An Overview of the Public’s Attitudes Towards AI and a Resource for Future Research. Social Science Research Network preprint 5108572. d...

  6. [14]

    Scheufele

    Dayeon Eom, Todd Newman, Dominique Brossard, and Dietram A. Scheufele. 2024. Societal guardrails for AI? Perspectives on what we know about public opinion on artificial intelligence.Science and Public Policy51, 5 (2024), 1004–1013. doi:10/qnz8

  7. [15]

    Kenny Davila, Srirangaraj Setlur, David Doermann, Bhargava Urala Kota, and Venu Govindaraju. 2021. Chart mining: A survey of methods for automated chart analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence43, 11 (2021), 3799–3819. doi:10/gh6rcz

  8. [16]

    Pengfei Hao, Hongqiu Wang, Guang Yang, and Lei Zhu. 2025. Enhancing visual reasoning with LLM-powered knowledge graphs for visual question localized-answering in robotic surgery.IEEE Journal of Biomedical and Health Informatics29, 12 (2025), 9027–9040. doi:10/qnz7

  9. [17]

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of LLM-based text-to-SQL.IEEE Transactions on Knowledge and Data Engineering37, 12 (2025), 7328–7345. doi:10/hbjqnp

  10. [18]

    Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. 2025. From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review. arXiv:2504.19678 doi:10/hbk46w

  11. [19]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al . 2025. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. InFindings of the Association for C...

  12. [20]

    Dominik Macháček and Peter Polák. 2025. Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025. arXiv:2506.17077 doi:10/hbk46x

  13. [21]

    Donggang Jia, Alexandra Irger, Lonni Besançon, Ondřej Strnad, Deng Luo, Johanna Björklund, Alexandre Kouyoumdjian, Anders Ynnerman, and Ivan Viola. 2025. VOICE: Visual oracle for interaction, conversation, and explanation.IEEE Transactions on Visualization and Computer Graphic...

  14. [22]

    Crime Animator

    Vahid Pooryousef, Lonni Besançon, Maxime Cordeil, Chris Flight, Alastair M. Ross AM, Richard Bassed, and Tim Dwyer. 2026. Criminator: An Easy-to-Use XR "Crime Animator" for Rapid Reconstruction and Analysis of Dynamic Crime Scenes. arXiv:2601.13689 doi:10/hbk465

  15. [23]

    Vahid Pooryousef, Maxime Cordeil, Lonni Besançon, Richard Bassed, and Tim Dwyer. 2024. Collaborative forensic autopsy documentation and supervised report generation using a hybrid mixed-reality environment and generative AI.IEEE Transactions on Visualization and Computer Graph...

  16. [24]

    Miriah Meyer, Bang Wong, Mark Styczynski, Tamara Munzner, and Hanspeter Pfister. 2010. Pathline: A tool for comparative functional genomics. Computer Graphics Forum29, 3 (2010), 1043–1052. doi:10/bw52z8

  17. [25]

    Patton, Luke Harmon, Alexander Lex, and Miriah Meyer

    Jen Rogers, Austin H. Patton, Luke Harmon, Alexander Lex, and Miriah Meyer. 2021. Insights from experiments with rigor in an evoBio design study.IEEE Transactions on Visualization and Computer Graphics27, 2 (2021), 1106–1116. doi:10/ghgt5p

  18. [26]

    Seyed Mahmoud Sajjadi Mohammadabadi, Burak Cem Kara, Can Eyupoglu, Can Uzay, Mehmet Serkan Tosun, and Oktay Karakuş. 2025. A survey of large language models: Evolution, architectures, adaptation, benchmarking, applications, challenges, and societal implications.Electronics14, ...

  19. [27]

    Vahid Pooryousef, Tim Dwyer, Richard Bassed, Maxime Cordeil, and Lonni Besançon. 2025. Lessons Learnt from Expert-Centred Studies Exploring Opportunities and Challenges for Immersive Forensic Investigation. arXiv:2512.15220 doi:10/hbk464

  20. [28]

    Michael Sedlmair, Miriah Meyer, and Tamara Munzner. 2012. Design study methodology: Reflections from the trenches and the stacks.IEEE Transactions on Visualization and Computer Graphics18, 12 (2012), 2431–2440. doi:10/f4fv7x

  21. [29]

    Tabalba, Christopher J

    Roderick S. Tabalba, Christopher J. Lee, Giorgio Tran, Nurit Kirshenbaum, and Jason Leigh. 2025. A pragmatics-based approach to proactive digital assistants for data exploration. InProceedings of the Conference on Conversational User Interfaces. ACM, New York, Article 69, 14 p...

  22. [30]

    Schlesener, Marcin Ziolkowski, Sai Keung Wong, Brent Westmoreland, and Sabarish V

    Elizabeth A. Schlesener, Marcin Ziolkowski, Sai Keung Wong, Brent Westmoreland, and Sabarish V. Babu. 2026. ‘Am I understood?’: How the interplay between embodiment and theory of mind behavior affects LLM-based conversational agents on perceived trust, anthropomorphism, presen...

  23. [31]

    Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, and Zhenyu Yan

  24. [32]

    Yulin Zhang, Cheng Shi, Yang Wang, and Sibei Yang. 2025. Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video. arXiv:2510.14560 doi:10/hbjqp6 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 Manuscript submitted to ACM

  25. [33]

    Manasi Vartak, Silu Huang, Tarique Siddiqui, Samuel Madden, and Aditya Parameswaran. 2017. Towards visualization recommendation systems. ACM SIGMOD Record45, 4 (2017), 34–39. doi:10/gfxrs9

  26. [35]

    arXiv:2505.14668 doi:10/hbk463 Presented at NeurIPS’25

    ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions. arXiv:2505.14668 doi:10/hbk463 Presented at NeurIPS’25

  27. [2018]

    InProceedings of the Joint Symposium on Computational Aesthetics , Sketch-Based Interfaces and Modeling, and Non-Photorealistic Animation and Rendering

    Reducing affective responses to surgical images through color manipulation and stylization. InProceedings of the Joint Symposium on Computational Aesthetics , Sketch-Based Interfaces and Modeling, and Non-Photorealistic Animation and Rendering. ACM, New York, Article 4, 13 pag...

  28. [2025]

    doi:10/g9mc86

    How should the advancement of large language models affect the practice of science?Proceedings of the National Academy of Sciences122, 5, Article e2401227121 (2025), 11 pages. doi:10/g9mc86

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.