Pith. sign in

REVIEW 3 major objections 6 minor 72 references

ImprovMate: Multimodal AI Assistant for Improv Actor Training

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ImprovMate claims that an LLM-based multimodal assistant can generate coherent narrative cues for solo improv actors by automatically tracking plot, characters, and objects.

desk verdict A solid system paper for LLM-based solo improv training, with a plausible central claim and a careful pilot, but the quantitative motion-validation evidence is weaker than the framing suggests. read the letter →

arxiv 2506.23180 v1 pith:LE6GYTRJ submitted 2025-06-29 cs.HC

Riccardo Drago , Yotam Sechayk , Mustafa Doga Dogan , Andrea Sanna , Takeo Igarashi This is my paper
classification cs.HC
keywords improvisationtraininginteractivestorytellingmultimodalAIlargelanguagemodelsnarrativecoherencecreativesupporttoolspilotstudyactor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ImprovMate is a training tool that lets an actor improvise alone while an LLM plays scene partner, audience, and narrator. The paper's central claim is that GPT-4o can read the actor's body and voice from a webcam and microphone, generate narrative continuations and cues, and track characters, places, and objects so the actor does not have to. The system was designed from a formative study with 15 improvisers and includes random hints, plot twists, and two structured exercises that mirror traditional drills. A pilot with three experienced actors suggested they embraced the AI when it resembled live practice and found the key-point tracker reduced memory load. The contribution is a new application of LLM multimodal understanding to solo improv training.

What carries the argument

The load-bearing mechanism is the Story Manager, an LLM agent split into a Story Initializer and a Story Generator. The Initializer builds premise, character, and story opening; the Generator takes the actor's latest webcam frames (sampled at 1 FPS) and Whisper transcript, consults key-points tables of who, where, and objects established so far, and emits the next story segment with random twists such as new characters, objects, location changes, and time jumps. This loop is what lets the system automate narrative stimuli while tracking continuity. Support tools add generated hints (who, where, what), an optional narrator, and two exercises, 'Three Things' for quick reactions and 'Endings' for concluding stories, that mirror traditional improv drills.

What would settle it

Take the same improv premises and have actors perform them twice, once with ImprovMate's generated cues and once with a human scene partner; have blind raters score narrative coherence and actors report cognitive load, and if the AI condition does not match or beat the human condition, the claim that automated generation reduces cognitive load while sustaining narratives is falsified. In addition, re-run the motion-labeling test against human-written action labels instead of the Motion-X pipeline's LLM-generated annotations; if the similarity score drops sharply at 1 FPS, the sampling-rate justification weakens.

Watch

Extended reading notes

Core claim

The central claim is that a general-purpose multimodal LLM can stand in for the human partner and audience in improv training: it watches the actor's body and listens to their speech, infers intentions, emotions, and actions, and returns a story continuation that respects the established characters, places, and objects. The paper reports that the Story Manager's 1 FPS frame sampling is sufficient for motion labeling, that generated hints and plot twists feel like live audience or director suggestions, and that the key-point tracker lowers memory load, letting actors focus on creativity rather than memory. The pilot study with three experienced improvisers is presented as initial evidence that actors accept AI assistance when it mirrors traditional practice.

Load-bearing premise

The system's usefulness rests on the assumption that GPT-4o can reliably infer an actor's intentions, emotions, and movements from one webcam frame per second plus an audio transcript, and can turn those inferences into a coherent story continuation.

Editorial extensions

If this is right

  • If ImprovMate works as described, a solo actor can rehearse improvisation without a human partner, with the AI absorbing story-tracking and cue-giving duties.
  • No specialized motion-capture hardware would be required: a webcam sampled at one frame per second plus an audio transcript is enough for the LLM to follow the performance.
  • LLM randomness becomes a feature rather than a bug, because unexpected hints and plot twists are used to train reactivity the way audience suggestions do in live improv.
  • The key-point tracker and structured exercises are meant to lower cognitive load and train specific skills, so actors can practice story conclusions and quick thinking at low energy cost.
  • If accepted, this positions AI as a backstage training partner rather than an on-stage performer, opening a different design direction from prior AI-improv systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: if the same multimodal loop is reliable, it could transfer to other embodied-rehearsal settings, such as language learners practicing dialogue, speakers rehearsing talks, or trainees running social scripts, because the core requirement is the same: respond to live speech and movement with coherent narrative feedback.
  • My inference: the paper's motion-labeling evaluation compares GPT-4o descriptions with Motion-X ground truth that was itself produced by an automated LLM pipeline, so the reported similarity scores may partly reflect agreement between LLM-based systems; a human-labeled motion benchmark would be a stricter test.
  • My inference: the pilot's three-participant sample can establish feasibility, but the claim that AI practice develops the same spontaneity and adaptability as human interaction would need longitudinal comparison against traditional group rehearsal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents ImprovMate, a multimodal AI assistant for solo improv actor training. It uses GPT-4o to analyze webcam video frames sampled at 1 FPS and Whisper-transcribed speech, then generates narrative continuations, character/plot hints, key-point tracking, and two training exercises. The design is motivated by a formative study with 15 actors that yields four design goals, and the system is evaluated in a pilot study with three actors whose qualitative feedback is positive. A supplementary study compares GPT-4o motion labels to Motion-X dataset labels using sentence-embedding similarity. The central claim is that actors can offload narrative continuity tracking to the system and are receptive to AI tools that mirror traditional improv practices.

Significance. ImprovMate addresses a real gap: prior AI-improv work concentrates on live performance with human intervention, while a solo training tool based on current LLMs is a natural next step. The paper's strengths include a concrete user-grounded design process, a modular system architecture, and an appendix with the exact prompts used, which supports replicability. The three-participant pilot offers genuine qualitative insight and is consistent with the claim of receptiveness, but it provides no quantitative or controlled evidence for reduced cognitive load or improved narrative coherence. The technical evaluation of motion recognition is weak as a validation of the system's core inference. If the central claim is taken in its strong form—that the system reliably tracks plot and character so the actor can focus on creativity—the current evidence is insufficient.

major comments (3)
  1. [Section A.1, Tables 1–3] The supplementary motion-label evaluation cannot support the design decision in Section 4.1 to sample video at 1 FPS. The 'ground truth' labels in Motion-X were generated by a specialized pipeline that includes LLMs, so a high similarity score shows agreement between two LLM-based systems rather than correctness relative to human judgment. Section A.1 states 'Errors due to incorrect video recognition were excluded,' but no count or description of the excluded cases is provided, making the AVG/MED values in Tables 1–3 uninterpretable as success rates. The BERT/SentenceTransformer metric measures textual similarity, not whether the inferred action, intention, or emotion is actually correct. Because the 1 FPS sampling decision rests on this evaluation, the core inference pipeline of the system is not empirically secured.
  2. [Abstract, §5] The pilot study does not provide adequate evidence for the abstract's claim that ImprovMate allows actors to focus on creativity without keeping track of plot or character continuity. The three participants came from the same improv club that supplied the formative study, there was no baseline or control condition, and no objective measures of narrative coherence or cognitive load were collected. Positive participant quotes may reflect novelty effects, politeness, or the improv convention of accepting and extending any input, including broken ones. To support the central claim, the authors need to either temper the claim to what the data show (self-reported helpfulness in a small pilot) or add behavioral evidence such as independent ratings of story continuity, counts of continuity errors, or pre/post measures of cognitive load.
  3. [§3 and §4.1/Figure 1] The formative study reports that AI-generated images for story context received 70% negative ratings from participants (Section 3), yet the implemented system includes an AI-generated character image (Figure 1C) and the paper does not explain this discrepancy. This is a direct tension between the stated user-grounded design process and the delivered system. The authors should either justify why the feature was retained (e.g., by making it optional and explaining that the ethical concern is mitigated) or remove the feature image and clarify that AI-generated imagery is not part of the evaluated system.
minor comments (6)
  1. [Abstract] The phrase 'leverages LLMs as GPTs' is imprecise; GPT-4o is a specific model, not a class of LLMs. Suggest rewording to 'leverages a GPT-4o-based LLM.'
  2. [Section 3] The sentence '46% of participants preferred reactivity, 39% were more neutral and 15% favored narrative coherence' is followed by an ungrammatical transition, and the phrase '75% (age 35+) - 80% (25-35)' is ambiguous about what the percentages refer to.
  3. [Section A.2] The text alternates between 'fp_skip_ratio' and 'fps_skip_ratio' (Tables 1–3 use the latter); unify the terminology.
  4. [References] Reference [33] lists a SIGACCESS companion venue, but the cited RIPT paper was published in the DIS 2018 companion; correct the bibliographic data.
  5. [Appendix A.1] The exact SentenceTransformer/BERT model (e.g., model name or version) is not reported, so the similarity values in Tables 1–3 are not reproducible.
  6. [Section 5] The paper would benefit from an explicit limitations paragraph, including the small sample, the overlap with the formative-study participants, and the absence of a baseline or control condition.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims rest on independent user feedback and the paper's own prompt-based system; only minor non-load-bearing self-citations appear.

full rationale

Walking the derivation chain: the system's design goals come from a formative study with 15 actors; implementation uses GPT-4o and Whisper with prompts given in Appendix B; the pilot-study acceptance claim is grounded in independent participant feedback, not in any quantity fitted by the system. There is no fitted parameter that is later renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation: the 1 FPS sampling decision is justified by the paper's own supplementary interrogative study rather than by an equivalently-sourced prior claim. The appendix's Motion-X evaluation is a validity concern rather than circularity: comparing GPT-4o labels with Motion-X's LLM-generated annotations, while excluding recognition errors, weakens the evaluation's external validity, but GPT-4o's labels are not derived from the Motion-X labels and no parameter is fitted to that set, so no prediction reduces to its input by construction. Two minor self-citations appear ([11] by co-author Dogan and [36] by co-authors Yotam and Igarashi), but they are used only as background support in related work and are not load-bearing for the central claim. Under the stated hard rules, that yields score 2 rather than 0, but the paper's main derivation is self-contained and not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of an off-the-shelf multimodal LLM, on the validity of the motion-label evaluation benchmark, on the transferability of traditional improv exercises to an AI setting, and on the representativeness of a three-person pilot. None of these assumptions is independently verified in the paper, and the motion-label benchmark is partially circular because the ground truth was generated by an LLM-based pipeline.

assumptions (4)
  • domain assumption GPT-4o can reliably infer actor intentions, emotions, and movements from 1 FPS sampled webcam frames and Whisper transcripts, and use them to produce coherent story continuations.
    The entire Story Manager pipeline depends on this multimodal understanding; Section 4.1 describes sending 1 FPS frames to GPT-4o and audio to Whisper for transcription.
  • domain assumption Motion-X dataset labels are a valid ground truth for evaluating GPT-4o motion understanding.
    Appendix A compares GPT-4o descriptions to Motion-X labels, but Motion-X annotations were generated via a specialized pipeline that includes LLM, so the benchmark is not fully independent.
  • domain assumption Traditional improv exercises such as abrupt endings and quick-response prompts are effective training techniques, and AI versions of them retain that effectiveness.
    Design goals DG1-DG4 and the exercises in Section 4.1 are based on formative study participant opinions, not on measured training outcomes.
  • domain assumption Self-reported enthusiasm from three experienced improv actors predicts broader acceptance and training benefit.
    The pilot study in Section 5 uses three participants recruited from the same club, with no longitudinal measures or learning outcomes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ImprovMate: Multimodal AI Assistant for Improv Actor Training." pith.science (2026). https://pith.science/paper/LE6GYTRJ

@misc{pith2026250623180,
  author       = {Pith},
  title        = {Pith review of: ImprovMate: Multimodal AI Assistant for Improv Actor Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LE6GYTRJ}},
  note         = {Machine review of arXiv:2506.23180}
}
read the original abstract

Improvisation training for actors presents unique challenges, particularly in maintaining narrative coherence and managing cognitive load during performances. Previous research on AI in improvisation performance often predates advances in large language models (LLMs) and relies on human intervention. We introduce ImprovMate, which leverages LLMs as GPTs to automate the generation of narrative stimuli and cues, allowing actors to focus on creativity without keeping track of plot or character continuity. Based on insights from professional improvisers, ImprovMate incorporates exercises that mimic live training, such as abrupt story resolution and reactive thinking exercises, while maintaining coherence via reference tables. By balancing randomness and structured guidance, ImprovMate provides a groundbreaking tool for improv training. Our pilot study revealed that actors might embrace AI techniques if the latter mirrors traditional practices, and appreciate the fresh twist introduced by our approach with the AI-generated cues.

Figures

Figures reproduced from arXiv: 2506.23180 by the authors.

Figure 1
Figure 1. The core interaction loop within ImprovMate. An actor uses full-body movements and dialogue to stage an improvi￾sation: (A) motion input, (B) the premise for the AI-generated story, (C) AI-generated character based on the context given by the user (image AI-generated), (D) AI-generated suggestions to help improvise, (E) the tool to track key points in the story, (F) AI-generated story consistent with the performance… view at source ↗
Figure 2
Figure 2. Single step of execution: the actor stages an improv performance ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The experienced actor trying ImprovMate. Her reactions to what the story proposes are highlighted. (1-3) Actor using story mode. (4) Actor trying one of the exercises. Blue boxes refer to user input; orange boxes refer to LLM-generated content. in shy actors by letting them perform without audience pressure. All participants reported feeling entertained and creatively stimu￾lated, as if “improvising with friends” (r… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 50 canonical work pages

  1. [1]

    2024.Crafting Narratives: Real-Time Generative Storytelling through Tangible AI

    Shipra Balasubramani. 2024.Crafting Narratives: Real-Time Generative Storytelling through Tangible AI. Ph. D. Dissertation. OCAD University

    work page 2024

  2. [2]

    Nicholas David Bowman, Lyndsay Michalik Gratch, Dan Pacheco, and T Man- aka Chock. 2024. Immersive Storytelling as a Technology, a Practice, and an Experience. Media and Communication 12 (2024)

    work page 2024

  3. [3]

    Mathewson

    Boyd Branch, Piotr Mirowski, and Kory W. Mathewson. 2021. Collabora- tive Storytelling with Human Actors and AI Narrators. In Proceedings of the Twelfth International Conference on Computational Creativity, México City, Méx- ico (Virtual), September 14-18, 2021 , Andrés Gómez de Silva Garza, Tony Veale, Wendy Aguilar, and Rafael Pérez y Pérez (Eds.). Asso...

    work page 2021

  4. [4]

    Mathewson, Sophia Ppali, and Alexan- dra Covaci

    Boyd Branch, Piotr Mirowski, Kory W. Mathewson, Sophia Ppali, and Alexan- dra Covaci. 2024. Designing and Evaluating Dialogue LLMs for Co-Creative Improvised Theatre. CoRR abs/2405.07111 (2024). doi:10.48550/ARXIV.2405.07111 arXiv:2405.07111

    arXiv doi:10.48550/arxiv.2405.07111 2024

  5. [5]

    Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative research in psychology 3, 2 (2006), 77–101

    work page 2006

  6. [6]

    Ben Burtenshaw. 2020. AI Stories: An Interactive Narrative System for Children. CoRR abs/2011.04242 (2020). arXiv:2011.04242 https://arxiv.org/abs/2011.04242

    work page Pith review arXiv 2020

  7. [7]

    Marc Cavazza, Fred Charles, and Steven J. Mead. 2002. Character-Based Inter- active Storytelling. IEEE Intell. Syst. 17, 4 (2002), 17–24. doi:10.1109/MIS.2002. 1024747

    work page doi:10.1109/mis.2002 2002

  8. [8]

    Mengyu Chen, Marko Peljhan, and Misha Sra. 2024. ConnectVR: A Trigger-Action Interface for Creating Agent-based Interactive VR Stories. In IEEE Conference Virtual Reality and 3D User Interfaces, VR 2024, Orlando, FL, USA, March 16-21,

    work page 2024

Show all 72 references
  1. [9]

    Sharon Lynn Chu, Francis K. H. Quek, and Kumar Sridharamurthy. 2014. Ready...action!: a performative authoring system for children to create animated stories. In Proceedings of the 11th Conference on Advances in Computer Enter- tainment Technology, ACE ’14, Funchal, Madeira, P...

    2014

  2. [10]

    Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. 2018. Creative Writing with a Machine in the Loop: Case Studies on Slo- gans and Stories. In Proceedings of the 23rd International Conference on Intelligent User Interfaces, IUI 2018, Tokyo, Japan...

    2018

  3. [11]

    Mustafa Doga Dogan, Eric J Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaço, Johnny Lee, Mar Gonzalez-Franco, and David Kim. 2024. Augmented Object Intelligence with XR-Objects. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24)...

    2024

  4. [12]

    Dustin Freeman and Ravin Balakrishnan. 2016. Improv Remix: Mixed-Reality Video Manipulation Using Whole-Body Interaction to Extend Improvised Theatre. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems, DIS ’16, Brisbane, QLD, Australia, June 04 - 08, 2...

    2016

  5. [13]

    Terrell Glenn, Ananya Ipsita, Caleb Carithers, Kylie Peppler, and Karthik Ramani

  6. [14]

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Re...

    2024 doi arXiv

  7. [15]

    Harsh Kumar, Jonathan Vincentius, Ewan Jordan, and Ashton Anderson. 2024. Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking. CoRR abs/2410.03703 (2024). doi:10.48550/ARXIV. 2410.03703 arXiv:2410.03703

    2024 doi arXiv

  8. [16]

    Changyang Li, Wanwan Li, Haikun Huang, and Lap-Fai Yu. 2022. Interactive augmented reality storytelling guided by scene semantics. ACM Trans. Graph. 41, 4 (2022), 91:1–91:15. doi:10.1145/3528223.3530061

    2022

  9. [17]

    Jiayang Li, Jiale Li, and Yunsheng Su. 2024. A Map of Exploring Human Interac- tion Patterns with LLM: Insights into Collaboration and Creativity. In Artificial Intelligence in HCI - 5th International Conference, AI-HCI 2024, Held as Part of the 26th HCI International Conferen...

    2024 doi

  10. [18]

    Hui Liang, Jian Chang, Shujie Deng, Can Chen, Ruofeng Tong, and Jian-Jun Zhang. 2017. Exploitation of multiplayer interaction and development of virtual puppetry storytelling using gesture control and stereoscopic devices. Comput. Animat. Virtual Worlds 28, 5 (2017). doi:10.10...

    2017 doi

  11. [19]

    Hui Liang, Jian Chang, Ismail Khalid Kazmi, Jian-Jun Zhang, and Peifeng Jiao

  12. [20]

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2023. Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset. Advances in Neural Information Processing Systems (2023)

    2023

  13. [21]

    Brian Magerko, Peter Dohogne, and Chris DeLeon. 2011. Employing Fuzzy Con- cept for Digital Improvisational Theatre. In Proceedings of the Seventh AAAI Con- ference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE 2011, October 10-14, 2011, Stanford, Cal...

    2011

  14. [22]

    Mathewson and Piotr Mirowski

    Kory W. Mathewson and Piotr Mirowski. 2017. Improvised Theatre Alongside Artificial Intelligences. In Proceedings of the Thirteenth AAAI Conference on Artifi- cial Intelligence and Interactive Digital Entertainment (AIIDE-17), October 5-9, 2017, Snowbird, Little Cottonwood Can...

    2017

  15. [23]

    Mathewson and Piotr Mirowski

    Kory W. Mathewson and Piotr Mirowski. 2018. Improbotics: Exploring the Imitation Game Using Machine Intelligence in Improvised Theatre. InProceedings of the Fourteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE 2018, November 13-17,...

    2018

  16. [24]

    Mathewson, and Shakir Mohamed

    Piotr Mirowski, Juliette Love, Kory W. Mathewson, and Shakir Mohamed. 2024. A Robot Walks into a Bar: Can Language Models Serve as Creativity SupportTools for Comedy? An Evaluation of LLMs’ Humour Alignment with Comedians. In The 2024 ACM Conference on Fairness, Accountability...

    2024 doi

  17. [25]

    Piotr Mirowski and Kory Wallace Mathewson. 2019. Human Improvised Theatre Augmented with Artificial Intelligence. In Proceedings of the 2019 ACM SIGCHI Conference on Creativity and Cognition, C&C 2019, San Diego, CA, USA, June 23-26, 2019, Steven Dow, Mary Lou Maher, Andruid K...

    2019

  18. [26]

    Mathewson, Boyd Branch, Thomas Winters, Ben Ver- hoeven, and Jenny Elfving

    Piotr Mirowski, Kory W. Mathewson, Boyd Branch, Thomas Winters, Ben Ver- hoeven, and Jenny Elfving. 2020. Rosetta Code: Improv in Any Language. In Proceedings of the Eleventh International Conference on Computational Creativity, ICCC 2020, Coimbra, Portugal, September 7-11, 20...

    2020

  19. [27]

    Karl Toby Rosenberg, Rubaiat Habib Kazi, Li-Yi Wei, Haijun Xia, and Ken Perlin

  20. [28]

    Hanieh Shakeri, Carman Neustaedter, and Steve DiPaola. 2021. SAGA: Col- laborative Storytelling with GPT-3. In Companion Publication of the 2021 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW 2021, Virtual Event, USA, October 23-27, 2021 , Jer...

    2021

  21. [29]

    Malloch, and Derek Reilly

    Abbey Singh, Ramanpreet Kaur, Peter Haltner, Matthew Peachey, Mar González- Franco, Joseph W. Malloch, and Derek Reilly. 2021. Story CreatAR: a Toolkit for Spatially-Adaptive Augmented Reality Storytelling. In IEEE Virtual Reality and 3D User Interfaces, VR 2021, Lisbon, Portu...

    2021

  22. [30]

    Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu

    Ben Swanson, Kory W. Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. 2021. Story Centaur: Large Language Model Few Shot Learning as a Creative Writing Tool. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisti...

    2021 doi

  23. [31]

    In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA 2024, Honolulu, HI, USA, May 11-16, 2024 , Florian ’Floyd’ Mueller, Penny Kyburz, Julie R

    DrawTalking: Towards Building Interactive Worlds by Sketching and Speak- ing. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA 2024, Honolulu, HI, USA, May 11-16, 2024 , Florian ’Floyd’ Mueller, Penny Kyburz, Julie R. Williamson, and Co...

    2024

  24. [32]

    Tianyi Wang, Xun Qian, Fengming He, Xiyun Hu, Yuanzhi Cao, and Karthik Ramani. 2021. GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications. In UIST ’21: The 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Ev...

    2021

  25. [33]

    Tiffany Wun, Claire Mikalauskas, Kevin Ta, Joshua Horacsek, and Lora Oehlberg

  26. [34]

    Wangyu Xue, Chen Qian, Jiayi Wu, Yang Zhou, Wentao Liu, Ju Ren, Siming Fan, and Yaoxue Zhang. 2024. ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries. CoRR abs/2412.12675 (2024). doi:10.48550/ARXIV.2412.12675 arXiv:2412.12675

    2024 doi arXiv

  27. [35]

    David Thue, Vadim Bulitko, Marcia Spetch, and Eric Wasylishen. 2007. Interactive Storytelling: A Player Modelling Approach. In Proceedings of the Third Artificial Intelligence and Interactive Digital Entertainment Conference, June 6-8, 2007, Stan- ford, California, USA, Jonath...

    2007

  28. [36]

    Sechayk Yotam, A Penarska Gabriela, A Randsalu Isa, Arzate Cruz Christian, and Igarashi Takeo. 2024. MyStoryKnight: A Character-drawing Driven Storytelling System Using LLM Hallucinations. IPSJ Interaction (2024)

    2024

  29. [37]

    Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In IUI 2022: 27th International Conference on Intelligent User Interfaces, Helsinki, Finland, March 22 - 25, 2022 , Giulio Jacucci, Samuel Kaski, Cristina Conati,...

    2022

  30. [38]

    Chao Zhang, Cheng Yao, Jiayi Wu, Weijia Lin, Lijuan Liu, Ge Yan, and Fangtian Ying. 2022. StoryDrawer: A Child-AI Collaborative Drawing System to Support Children’s Creative Visual Storytelling. In CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA,...

    2022

  31. [39]

    Lei Zhang, Daekun Kim, Youjean Cho, Ava Robinson, Yu Jiang Tham, Rajan Vaish, and Andrés Monroy-Hernández. 2024. Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of Things. InProceedings of the CHI Conference on Human Factors in Computin...

    2024

  32. [40]

    Shuran Yang. 2023. Storytelling and user experience in the cultural metaverse. Heliyon 9, 4 (2023)

    2023

  33. [41]

    Yijun Zhao, Yiming Cheng, Shiying Ding, Yan Fang, Wei Cao, Ke Liu, and Jiacheng Cao. 2024. Magic Camera: An AI Drawing Game Supporting Instantaneous Story Creation for Children. In Proceedings of the 23rd Annual ACM Interaction Design and Children Conference, IDC 2024, Delft, ...

    2024

  34. [45]

    Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tong- shuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022. StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. In CHI ’22: CHI Conferenc...

    2022

  35. [47]

    The input object is a short improv performance that is to be used as a continuation for the narrative of the improvisation story

  36. [48]

    Analyze the key movements in the video , focusing on how the performer ' s movements interact with their spoken words or sounds in the audio

  37. [49]

    Consider how these movements connect with the improvisational flow and transform or enhance the narrative in real - time

  38. [50]

    Be faithful to the context and use the characters , places or actions mentioned

    As context for the improv performance , use the following hints to guide your analysis : { hints }. Be faithful to the context and use the characters , places or actions mentioned . - 'who ': he character that the performer is impersonating , and will be the protagonist of the...

  39. [51]

    Consider the premise of the story , it consist main scenario or conflict of the story : { premise }

  40. [52]

    Consider the narrative context that has been established so far : { story }

  41. [53]

    who ": The characters present in the story told so far . -

    Consider the key points that have been identified in the story : - " who ": The characters present in the story told so far . - " where ": The location where the story takes place . - " objects ": The objects present in the story told so far . - { keypoint }

  42. [54]

    who " ,

    Generate the next story part based on the context of the story , considering description and emotion of the improv performance . The next story part should be : - Not more than { length } sentences . - Take into account the " who " , " where " and " objects " present in the st...

  43. [55]

    Understand the story so far . ImprovMate: Multimodal AI Assistant for Improv Actor Training DIS ’25 Companion, July 5–9, 2025, Funchal, Portugal fps_skip_ratio AVG MED STD AVG TKN CMP AVG TKN PMT AVG TKN TOT Context - 0.5 0.712 0.723 0.156 92.484 2147.979 2240.463 NoContext - ...

    2025

  44. [56]

    Help me generate { number } unique actions the main character may perform

  45. [57]

    Each action should advance the current story somehow

  46. [58]

    - Description , very short paragraph with more details

    Action is defined by : - Title , few words describing the action . - Description , very short paragraph with more details . B.3 Generate Story Part This prompt helps the LLM continue the story in a dynamic way, incorporating surprise elements such as plot twists, time skips, o...

  47. [59]

    Understand the input object

  48. [60]

    Understand the story so far

  49. [61]

    Continue the story based on the main character performing the given action

  50. [62]

    - Do not name the main character

    Generate a short visual description of a key moment in the new part : - Describe the environment . - Do not name the main character . B.4 Hints This prompt is used to generate creative hint combinations that serve as the initial inspiration for improv scenes. These story seeds...

  51. [63]

    Generate { number } elements , each composed of 3 fields , the first answering the question ' Who ?', the second ' Where ? ' and the third ' What happened ? '

  52. [64]

    ( examples : a clown , a turtle , the Pope )

    The answer to ' Who ? ' should be a character that can be used as a protagonist . ( examples : a clown , a turtle , the Pope )

  53. [65]

    The answer to ' Where ? ' should be a location where the story takes place

  54. [66]

    B.5 Endings This prompt generates short, open-ended story beginnings

    The answer to ' What happened ? ' should be a short event that can be used as the starting point of the story . B.5 Endings This prompt generates short, open-ended story beginnings. These openings are presented to the user as challenges to conclude in a single performance. You...

  55. [67]

    - Setting : Describe where the story takes place , incorporating vivid details

    Create an original story introduction which includes : - Character : Introduce a main character with a few unique traits . - Setting : Describe where the story takes place , incorporating vivid details . - Event : Describe an unusual or intriguing situation that the character ...

  56. [68]

    Develop the story so that it sets up a decision point or situation the character must respond to , without concluding the story

  57. [69]

    B.6 Three Things This prompt creates short, engaging questions that require users to quickly list three creative or humorous items or ideas

    Generate a visual description of a key moment in this part , capture the atmosphere and scene details . B.6 Three Things This prompt creates short, engaging questions that require users to quickly list three creative or humorous items or ideas. You are a creative assistant hel...

  58. [70]

    3 things

    Generate unique and engaging questions that prompt quick , imaginative responses . Each question should : - Be open - ended and encourage creativity . - Challenge players to think of "3 things " or similar sets , such as "3 items " or "3 ways ." - Cover a mix of themes , inclu...

  59. [71]

    Questions should be short , clear , and easy to understand

  60. [72]

    3 ways to climb up the stairs

    Ensure a variety of themes across the questions , such as : - Actions : "3 ways to climb up the stairs ..." - Items : "3 things you would bring to the moon ..." - Phrases : "3 things to say at a funeral ..." - Emotions : "3 ways to show someone you care ..."

  61. [2015]

    Puppet Narrator: Utilizing Motion Sensing Technology in Storytelling for Young Children. In 7th International Conference on Games and Virtual Worlds for Serious Applications, VS-Games 2015, Skövde, Sweden, September 16-18, 2015 , Per Backlund, Henrik Engström, and Fotis Liarok...

    2015

  62. [2018]

    RIPT: Improvising with an Audience-Sourced Performance Robot. In Companion Publication of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, DIS 2018, Hong Kong, China, June 09-13, 2018 , Ilpo Koskinen, Youn-Kyung Lim, Teresa Cerratto-Pargman, Kenn...

    2018

  63. [2020]

    StoryMakAR: Bringing Stories to Life With An Augmented Reality & Physi- cal Prototyping Toolkit for Youth. InCHI ’20: CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, April 25-30, 2020 , Regina Bernhaupt, Flo- rian ’Floyd’ Mueller, David Verweij, Josh A...

    2020

  64. [2024]

    doi:10.1109/VR58804.2024.00051 DIS ’25 Companion, July 5–9, 2025, Funchal, Portugal Drago, et al

    IEEE, 286–297. doi:10.1109/VR58804.2024.00051 DIS ’25 Companion, July 5–9, 2025, Funchal, Portugal Drago, et al

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.