REVIEW 3 major objections 5 minor 36 references
Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read LLM planetarium pilots can cut human workload as co-pilots, but cannot replace pilots.
desk verdict An honest little pilot study with a useful error taxonomy, but the headline workload-reduction claim rests on an extrapolation, not on the co-pilot condition the paper actually proposes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the conversational AI pilot built on the astrophysics visualization software OpenSpace and a low-latency multimodal language model. It receives the guide's speech (in reactive mode through a hand-triggered microphone, in proactive mode through streaming speech-to-text), interprets the speech as commands, and dispatches tool calls through OpenSpace's Lua API to travel between scene nodes, toggle asset visibility, change simulation time, or do nothing via a 'no-op' call. The comparison between reactive and proactive modes is the experimental pivot.
What would settle it
Run real public planetarium shows with the same AI co-pilot and measure human pilot workload and experimenter interventions; if the AI requires intervention nearly as often as it saves actions, or if guides report no workload drop, the central claim fails. A sharper version would compare AI co-pilot shows head-to-head with human-human pilot pairs on the same measures.
Extended reading notes
Core claim
The paper's central claim is that an LLM-based pilot, which listens to a guide's speech and executes commands like moving the camera, changing simulation time, and toggling assets, is not able to replace a human pilot in a live planetarium show. The decisive observation is that the AI is good at toggling assets and preparing actions but poor at camera control and pacing, which demand temporal awareness and precision. The paper therefore proposes a division of labor: keep a human pilot for camera and pacing, and let the AI act as a co-pilot to reduce cognitive load and enable multitasking. Five guides tested reactive and proactive modes; proactive felt more natural but less reliable, while re
Load-bearing premise
The conclusion that AI co-pilots reduce workload rests on the assumption that the experience of five guides in simulated shows—where the experimenter had to intervene repeatedly because the software was unstable—and the expectations of two interviewed experts predict what would happen in real public shows.
Editorial extensions
If this is right
- Planetarium shows would keep a human pilot for camera control and pacing, while the AI co-pilot handles toggling and preparing visual assets.
- Proactive listening, once reliability improves, can reduce mental load and preserve narrative flow because guides no longer need to press a trigger.
- The error dimensions identified—detection, reasoning, context, and naturalness—offer a concrete checklist for building and evaluating AI pilots.
- AI could assist during show preparation and onboarding, letting novice pilots assemble and test storytelling blocks in a low-risk setting.
Reading between the lines
- If the co-pilot division of labor holds, the same pattern could transfer to other live visualization settings, such as surgery or forensic reconstruction, where a human controls continuous motion while AI preloads context.
- The paper's 'interaction recommender system' idea suggests a less autonomous intermediate step: the AI suggests actions rather than executing them, which may be more robust before streaming speech and latency improve.
- The error dimension spectrum could be formalized into a benchmark with human-coded ground truth, making future AI pilot comparisons quantitative rather than interview-based.
- A reading of the mixed proactive/reactive preference: reliability is the gating factor; if proactive mode's reliability approaches reactive mode's, its fluidity advantage may make it the default.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the design and evaluation of an LLM-based conversational pilot for the OpenSpace planetarium visualization system, intended to assist guides during live dome shows. The system operates in two modes: a reactive mode in which the guide triggers voice commands with a handheld microphone, and a proactive mode in which the system continuously listens and executes implicit commands. Five professional guides from the authors' institution conducted simulated shortened shows in a real dome, and two additional experts provided interview-based descriptions of human piloting practice. The paper reports qualitative thematic analysis and an error-case taxonomy. The central claim is that, while AI pilots are not yet able to replace human pilots, they could serve as co-pilots to reduce human pilot workload and enable multitasking.
Significance. If the central claim were empirically supported, the work would be a useful early contribution to an emerging area: LLM-driven interaction with visualization in high-stakes, time-critical, public-facing settings. The paper's strengths are its concrete system implementation, the comparison of reactive vs. proactive conversational modes, the honest reporting of failures and mixed participant preferences, and the articulation of an error taxonomy with concrete failure examples. The paper also identifies a genuinely under-explored design direction, interaction recommender systems, that could inform future work. However, the significance is currently limited because the headline conclusion about workload reduction is an extrapolation from subjective impressions and speculative participant statements rather than a measured outcome of the study's design.
major comments (3)
- [§5.3 and overall design (RQ1)] The central claim that AI can act as a co-pilot to reduce human workload and enable multitasking is not directly tested. The study compares two AI-only conditions (reactive and proactive), in which the guide is the sole pilot and the AI executes commands; there is no condition in which a human pilot works alongside the AI. The workload-reduction conclusion rests on participants' speculative statements (e.g., P3's satellite anecdote in §5.3) and on a human-pilot baseline reconstructed from two expert interviews, not on any measured comparison. RQ1 is phrased as 'AI co-pilots compared to human co-pilots,' but the experiment never instantiates the co-pilot configuration. To support the claim, the paper needs either a condition with human+AI teaming, or a substantial reframing of the conclusion as a hypothesis for future work rather than a study result.
- [§4.2 and §4.3] The evidence base for comparing AI piloting with human piloting is fragile. The human-pilot baseline is based on interviews with two consulting experts (C1, C2) rather than on logged or observed human pilot performance. In the AI conditions, the first author had to intervene several times because of software instability, and P4 could not run the system at all, instead watching P5's session. The paper states in §4.3 that 'at this stage, we focus on reporting on the insights from the interviews' and defers analysis of the collected logs. Consequently, the reported comparisons are qualitative impressions from a small, partly interrupted sample, and the abstract's 'results show' phrasing overstates the evidentiary strength. The paper should clearly label the workload-reduction and co-pilot statements as preliminary hypotheses and temper the abstract accordingly.
- [§5.4 and §4.3] The error-case analysis is presented as a substantive result, but it is anecdotal. Section 4.3 says the system logs contain latency, number, and success rate of interventions, yet §4.3 also says the paper focuses on interview insights, and §5.4 explicitly states that a 'qualitative analysis of error cases (both their characteristics, and prevalence) is needed.' No counts, success rates, or inter-rater coding are reported. The four error dimensions are plausible and useful as a taxonomy, but they are not derived from a systematic coding of the logs. Since the paper claims an 'error-case analysis from the system operation logs' in §4, this inconsistency should be resolved—either by reporting quantitative error data or by explicitly labeling the taxonomy as a set of observed examples rather than an analysis.
minor comments (5)
- [Abstract and §4.1] The abstract says '7 professional guides,' but §4.1 describes five expert participants (P1–P5) and two consulting experts (C1, C2). The two experts did not use the system. Please clarify the wording to distinguish study participants from expert interviewees.
- [§4.2] P4 could not run the system and instead observed P5; the subsequent joint interview means the effective number of independent system-use sessions is four, not five. This should be stated explicitly in the participants and protocol description.
- [§5.4] The sentence 'A qualitative analysis of error cases (both their characteristics, and prevalence) is needed' is internally inconsistent: prevalence is a quantitative measure. Moreover, 'is needed' suggests the analysis has not been performed, which conflicts with the earlier claim of an 'error-case analysis from the system operation logs' in §4. Please revise to avoid this contradiction.
- [Title page and references] The ACM reference format section contains placeholder dates ('February 2018', 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009') and the author list has inconsistent spacing in the second author's name ('M UJTABA FADHIL JA W AD'). These formatting issues should be corrected.
- [§5.2] The finding that the reactive mode 'added to the mental load' is reported without a direct comparison of measured cognitive load. Since cognitive load is a central concept in the paper's argument, it would help to define how the authors infer cognitive load from the interviews and to acknowledge that this is a perceived, not measured, construct.
Circularity Check
No circularity: the paper's conclusions are qualitative inductions from interviews, not results forced by definitions, fits, or self-citations.
full rationale
The paper makes no formal derivation or quantitative prediction; its central claim that AI pilots 'could become useful as co-pilots to reduce workload of human pilots and allow multitasking' is an inductive synthesis of participant interviews (Section 5.3) and expert comparisons (Section 5.1), not a quantity computed from the system. No parameter is fitted and then reported as a prediction; no 'uniqueness theorem' or prior claim by the same authors is used to force the conclusion. The self-citations present (e.g., [6] for the system's inspiration, [7] for the state of the art, [18] for prior proactive-agent work) position the work within a lineage but do not carry the conclusion. The manuscript itself flags limitations that reduce evidentiary strength but not circularity: Section 4.2 states 'Due to some instability with the software, the first author had to intervene several times during the presentation' and P4's run failed, and Section 4.3 says 'At this stage, we focus on reporting on the insights from the interviews.' These are reliability/generalizability threats, not cases where the outcome equals an input by construction.
Assumptions & free parameters
assumptions (3)
- domain assumption Professional guides' qualitative feedback is representative of the broader population of planetarium guides.
- domain assumption The simulated shortened dome show is a valid proxy for a real public show.
- domain assumption Human piloting baseline can be captured from two consulting experts' interviews rather than direct measurement.
Cite this review
Pith. "Pith review of Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers." pith.science (2026). https://pith.science/paper/NEZPI6OF
@misc{pith2026260120466,
author = {Pith},
title = {Pith review of: Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers},
year = {2026},
howpublished = {\url{https://pith.science/paper/NEZPI6OF}},
note = {Machine review of arXiv:2601.20466}
}
read the original abstract
We designed and evaluated an AI pilot in a planetarium visualization software, OpenSpace, for public shows in science centers. The piloting role is usually given to a human working in close collaboration with the guide on stage. We recruited 7 professional guides with extensive experience in giving shows to the public to study the impact of the AI-piloting on the overall experience. The AI-pilot is a conversational AI-agent listening to the guide and interpreting the verbal statements as commands to execute camera motions, change simulation time, or toggle visual assets. Our results show that, while AI pilots lack several critical skills for live shows, they could become useful as co-pilots to reduce workload of human pilots and allow multitasking. We propose research directions toward implementing visualization pilots and co-pilots in live settings.
Reference graph
Works this paper leans on
-
[1]
Basole and Timothy Major
Rahul C. Basole and Timothy Major. 2024. Generative AI for visualization: Opportunities and challenges.IEEE Computer Graphics and Applications 44, 2 (2024), 55–64. doi:10/nm2j Manuscript submitted to ACM Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers 7
2024
-
[2]
Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Marc Soubeyrand, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic
Lonni Besançon, Amir Semmo, David J. Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Marc Soubeyrand, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic. 2020. Reducing affective responses to surgical images and videos through stylization.Computer Graphics Forum39, 1 (2020), 462–483. doi:10/gjbm72
2020
-
[3]
Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic
Lonni Besançon, Amir Semmo, David J. Biau, Bruno Frachet, Virginie Pineau, El Hadi Sariali, Rabah Taouachi, Tobias Isenberg, and Pierre Dragicevic
-
[4]
Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D
Marcel Binz, Stephan Alaniz, Adina Roskies, Balazs Aczel, Carl T. Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D. West, Qiong Zhang, Richard M. Shiffrin, Samuel J. Gershman, Vencislav Popov, Emily M. Bender, Marco Marelli, Matthew M. Botvinick, Zeynep Akata, and Eric Schulz
-
[5]
Alexander Bock, Emil Axelsson, Jonathas Costa, Gene Payne, Micah Acinapura, Vivian Trakinski, Carter Emmart, Claudio Silva, Charles Hansen, and Anders Ynnerman. 2020. OpenSpace: A system for astrographics.IEEE Transactions on Visualization and Computer Graphics26, 1 (2020), 633–642. doi:10/ggsrbm
2020
-
[6]
Mathis Brossier, Alexander Bock, Konrad Schönborn, Tobias Isenberg, Anders Ynnerman, and Lonni Besançon. 2024. In space, no one (but AI) can hear you scream. InPosters at IEEE VIS. doi:10/hbk46z
2024
-
[7]
Mathis Brossier, Tobias Isenberg, Konrad Schönborn, Jonas Unger, Mario Romero, Johanna Björklund, Anders Ynnerman, and Lonni Besançon. 2026. State of the Art of LLM-Enabled Interaction with Visualization. arXiv:2601.14943 doi:10/hbk462
arXiv 2026
-
[8]
Mathis Brossier, Robin Skånberg, Lonni Besançon, Mathieu Linares, Tobias Isenberg, Anders Ynnerman, and Alexander Bock. 2023. Moliverse: Contextually embedding the microcosm into the universe.Computers & Graphics112 (2023), 22–30. doi:10/gr7bbz
2023
Show all 36 references
-
[9]
Shubham Chatterjee, Xi Wang, Shuo Zhang, Sajad Ebrahimi, Zhaochun Ren, Debasis Ganguly, Gareth Jones, Emine Yilmaz, and Hamed Zamani
-
[10]
Juntong Chen, Jiang Wu, Jiajing Guo, Vikram Mohanty, Xueming Li, Jorge Piazentin Ono, Wenbin He, Liu Ren, and Dongyu Liu. 2025. InterChat: Enhancing generative visual analytics using multimodal interactions.Computer Graphics Forum44, 3, Article e70112 (2025), 24 pages. doi:10/g9qr4q
2025
-
[11]
Zina Chkirbene, Ridha Hamila, Ala Gouissem, and Unal Devrim. 2024. Large language models (LLM) in industry: A survey of applications, challenges, and trends. InProceedings of the International Conference on Smart Communities: Improving Quality of Life using AI, Robotics and Io...
2024
-
[12]
InProceedings of the International Conference on Information and Knowledge Management
ProActLLM: Proactive conversational information seeking with large language models. InProceedings of the International Conference on Information and Knowledge Management. ACM, New York, 6894–6897. doi:10/qn2b
-
[13]
2025.What Does the Public Think About AI? An Overview of the Public’s Attitudes Towards AI and a Resource for Future Research
Noemi Dreksler, Harry Law, Chloe Ahn, Daniel Schiff, Kaylyn Jackson Schiff, and Zachary Peskowitz. 2025.What Does the Public Think About AI? An Overview of the Public’s Attitudes Towards AI and a Resource for Future Research. Social Science Research Network preprint 5108572. d...
2025
-
[14]
Scheufele
Dayeon Eom, Todd Newman, Dominique Brossard, and Dietram A. Scheufele. 2024. Societal guardrails for AI? Perspectives on what we know about public opinion on artificial intelligence.Science and Public Policy51, 5 (2024), 1004–1013. doi:10/qnz8
2024
-
[15]
Kenny Davila, Srirangaraj Setlur, David Doermann, Bhargava Urala Kota, and Venu Govindaraju. 2021. Chart mining: A survey of methods for automated chart analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence43, 11 (2021), 3799–3819. doi:10/gh6rcz
2021
-
[16]
Pengfei Hao, Hongqiu Wang, Guang Yang, and Lei Zhu. 2025. Enhancing visual reasoning with LLM-powered knowledge graphs for visual question localized-answering in robotic surgery.IEEE Journal of Biomedical and Health Informatics29, 12 (2025), 9027–9040. doi:10/qnz7
2025
-
[17]
Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of LLM-based text-to-SQL.IEEE Transactions on Knowledge and Data Engineering37, 12 (2025), 7328–7345. doi:10/hbjqnp
2025
-
[18]
Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. 2025. From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review. arXiv:2504.19678 doi:10/hbk46w
2025 arXiv
-
[19]
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al . 2025. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. InFindings of the Association for C...
2025
-
[20]
Dominik Macháček and Peter Polák. 2025. Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025. arXiv:2506.17077 doi:10/hbk46x
2025 arXiv
-
[21]
Donggang Jia, Alexandra Irger, Lonni Besançon, Ondřej Strnad, Deng Luo, Johanna Björklund, Alexandre Kouyoumdjian, Anders Ynnerman, and Ivan Viola. 2025. VOICE: Visual oracle for interaction, conversation, and explanation.IEEE Transactions on Visualization and Computer Graphic...
2025
-
[22]
Crime Animator
Vahid Pooryousef, Lonni Besançon, Maxime Cordeil, Chris Flight, Alastair M. Ross AM, Richard Bassed, and Tim Dwyer. 2026. Criminator: An Easy-to-Use XR "Crime Animator" for Rapid Reconstruction and Analysis of Dynamic Crime Scenes. arXiv:2601.13689 doi:10/hbk465
2026
-
[23]
Vahid Pooryousef, Maxime Cordeil, Lonni Besançon, Richard Bassed, and Tim Dwyer. 2024. Collaborative forensic autopsy documentation and supervised report generation using a hybrid mixed-reality environment and generative AI.IEEE Transactions on Visualization and Computer Graph...
2024
-
[24]
Miriah Meyer, Bang Wong, Mark Styczynski, Tamara Munzner, and Hanspeter Pfister. 2010. Pathline: A tool for comparative functional genomics. Computer Graphics Forum29, 3 (2010), 1043–1052. doi:10/bw52z8
2010
-
[25]
Patton, Luke Harmon, Alexander Lex, and Miriah Meyer
Jen Rogers, Austin H. Patton, Luke Harmon, Alexander Lex, and Miriah Meyer. 2021. Insights from experiments with rigor in an evoBio design study.IEEE Transactions on Visualization and Computer Graphics27, 2 (2021), 1106–1116. doi:10/ghgt5p
2021
-
[26]
Seyed Mahmoud Sajjadi Mohammadabadi, Burak Cem Kara, Can Eyupoglu, Can Uzay, Mehmet Serkan Tosun, and Oktay Karakuş. 2025. A survey of large language models: Evolution, architectures, adaptation, benchmarking, applications, challenges, and societal implications.Electronics14, ...
2025
-
[27]
Vahid Pooryousef, Tim Dwyer, Richard Bassed, Maxime Cordeil, and Lonni Besançon. 2025. Lessons Learnt from Expert-Centred Studies Exploring Opportunities and Challenges for Immersive Forensic Investigation. arXiv:2512.15220 doi:10/hbk464
2025
-
[28]
Michael Sedlmair, Miriah Meyer, and Tamara Munzner. 2012. Design study methodology: Reflections from the trenches and the stacks.IEEE Transactions on Visualization and Computer Graphics18, 12 (2012), 2431–2440. doi:10/f4fv7x
2012
-
[29]
Tabalba, Christopher J
Roderick S. Tabalba, Christopher J. Lee, Giorgio Tran, Nurit Kirshenbaum, and Jason Leigh. 2025. A pragmatics-based approach to proactive digital assistants for data exploration. InProceedings of the Conference on Conversational User Interfaces. ACM, New York, Article 69, 14 p...
2025
-
[30]
Schlesener, Marcin Ziolkowski, Sai Keung Wong, Brent Westmoreland, and Sabarish V
Elizabeth A. Schlesener, Marcin Ziolkowski, Sai Keung Wong, Brent Westmoreland, and Sabarish V. Babu. 2026. ‘Am I understood?’: How the interplay between embodiment and theory of mind behavior affects LLM-based conversational agents on perceived trust, anthropomorphism, presen...
2026
-
[31]
Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, and Zhenyu Yan
-
[32]
Yulin Zhang, Cheng Shi, Yang Wang, and Sibei Yang. 2025. Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video. arXiv:2510.14560 doi:10/hbjqp6 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 Manuscript submitted to ACM
2025
-
[33]
Manasi Vartak, Silu Huang, Tarique Siddiqui, Samuel Madden, and Aditya Parameswaran. 2017. Towards visualization recommendation systems. ACM SIGMOD Record45, 4 (2017), 34–39. doi:10/gfxrs9
2017
-
[35]
arXiv:2505.14668 doi:10/hbk463 Presented at NeurIPS’25
ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions. arXiv:2505.14668 doi:10/hbk463 Presented at NeurIPS’25
-
[2018]
InProceedings of the Joint Symposium on Computational Aesthetics , Sketch-Based Interfaces and Modeling, and Non-Photorealistic Animation and Rendering
Reducing affective responses to surgical images through color manipulation and stylization. InProceedings of the Joint Symposium on Computational Aesthetics , Sketch-Based Interfaces and Modeling, and Non-Photorealistic Animation and Rendering. ACM, New York, Article 4, 13 pag...
-
[2025]
doi:10/g9mc86
How should the advancement of large language models affect the practice of science?Proceedings of the National Academy of Sciences122, 5, Article e2401227121 (2025), 11 pages. doi:10/g9mc86
2025
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.