pith. sign in

arxiv: 2410.22177 · v2 · submitted 2024-10-29 · 💻 cs.HC · cs.AI

Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes

Junlong Chen , Jens Grubert , Per Ola Kristensson This is my paper

Pith reviewed 2026-05-23 18:39 UTC · model grok-4.3

classification 💻 cs.HC cs.AI
keywords LLM-assisted 3D editingimmersive environmentsuser studyinteraction patternsmultimodal interfacesnatural language interfaces3D scene manipulationdesign recommendations
0
0 comments X

The pith

Empirical study with 12 participants demonstrates that LLM-assisted systems support productive 3D scene manipulation in immersive environments.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper reports an empirical user study that examines how people use large language models to edit 3D scenes while immersed in virtual environments. It pairs quantitative logs of participant actions with questionnaire responses to surface recurring interaction patterns and the main obstacles users face. A reader would care because new applications that combine LLMs with 3D content are appearing, and concrete evidence about workable interaction styles can shape more usable tools. The work concludes that such systems already allow productive use while pointing to concrete improvements for natural language components of 3D design software.

Core claim

Through an empirical study with 12 participants, the authors show that LLM-assisted interactive systems can be used productively in immersive environments for 3D scene manipulation, while also mapping out common interaction patterns and key barriers that inform design recommendations for future systems.

What carries the argument

The empirical user study that combines quantitative usage data with post-experience questionnaire feedback to expose interaction patterns and barriers.

Load-bearing premise

The interaction patterns and barriers observed with a sample of 12 participants are representative of broader user behavior when manipulating 3D scenes with LLMs in immersive settings.

What would settle it

A larger follow-up study that records substantially different prompting styles, error rates, or productivity measures across a wider participant pool would undermine the generalizability of the observed patterns.

Figures

Figures reproduced from arXiv: 2410.22177 by Jens Grubert, Junlong Chen, Per Ola Kristensson.

Figure 1
Figure 1. Figure 1: Example workflow for scene editing with our proposed A [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Workflow of the ASSISTVR system designed for the study. In the training phase, only Azure Conversational Language Understanding (CLU) is involved. The developer labels a number of utterances with intents and entities, and finetunes the Azure CLU model. The model is iteratively improved based on performance metrics. In the deployment phase, Azure CLU classifies user speech input into different intents. If t… view at source ↗
Figure 3
Figure 3. Figure 3: Example of the draggable panel. The panel shows that [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Number of remaining elemental editing steps to match the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Interaction strategies adopted by different users across the duration of Task 1 (top left) and Task 2 (top right). Time on the horizontal axis [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Box plots of the time spent in minutes for all participants on the incremental exploration (IE) strategy and the bulk modification (BM) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
read the original abstract

As more applications of large language models (LLMs) for 3D content for immersive environments emerge, it is crucial to study user behaviour to identify interaction patterns and potential barriers to guide the future design of immersive content creation and editing systems which involve LLMs. In an empirical user study with 12 participants, we combine quantitative usage data with post-experience questionnaire feedback to reveal common interaction patterns and key barriers in LLM-assisted 3D scene editing systems. We identify opportunities for improving natural language interfaces in 3D design tools and propose design recommendations for future LLM-integrated 3D content creation systems. Through an empirical study, we demonstrate that LLM-assisted interactive systems can be used productively in immersive environments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper presents a mixed-methods empirical user study with 12 participants that combines quantitative usage logs and post-experience questionnaires to analyze interaction patterns and barriers when using LLM-assisted systems for manipulating 3D scenes in immersive environments. It identifies common strategies, key challenges, and offers design recommendations for future LLM-integrated 3D content creation tools, with the central claim that such systems can be used productively in immersive settings.

Significance. If the observational findings hold, the work provides an early empirical foundation for understanding user behaviors in LLM-assisted immersive 3D editing, which is timely given the emergence of such applications. The mixed-methods approach (logs + questionnaires) is appropriate for an exploratory HCI study and directly supports the feasibility demonstration.

major comments (2)
  1. [User Study / Methodology] User Study / Methodology: The central productivity and pattern-identification claims rest on data from only 12 participants with no reported statistical tests, power analysis, exclusion criteria, or demographic details; this leaves the evidence for 'common' patterns and productive use descriptive rather than robust, directly weakening the generalizability asserted in the abstract and discussion.
  2. [Results / Analysis] Results / Analysis: The quantitative usage data and questionnaire feedback are presented without baseline comparisons, effect sizes, or inferential statistics, so the claim that LLM-assisted systems 'can be used productively' is supported only by existence (successful sessions occurred) rather than by evidence that the observed patterns exceed chance or prior non-LLM interfaces.
minor comments (2)
  1. [Abstract / Introduction] The abstract and introduction could more explicitly distinguish the feasibility claim from the pattern-identification claims to clarify what the small sample can and cannot support.
  2. [Figures / Tables] Figure captions and table descriptions should include exact sample sizes and any filtering applied to the log data for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback on our manuscript. We value the referee's assessment of the study's timeliness and methodological appropriateness for an exploratory HCI investigation. We address each major comment below, providing clarifications on the study's design and scope.

read point-by-point responses

  1. Referee: [User Study / Methodology] User Study / Methodology: The central productivity and pattern-identification claims rest on data from only 12 participants with no reported statistical tests, power analysis, exclusion criteria, or demographic details; this leaves the evidence for 'common' patterns and productive use descriptive rather than robust, directly weakening the generalizability asserted in the abstract and discussion.

    Authors: This study is explicitly positioned as an exploratory investigation into user behaviors with LLM-assisted 3D manipulation in immersive environments, following established practices in HCI for early-stage research on novel interfaces. Small participant numbers (N=12) are common in such studies to uncover qualitative patterns and barriers rather than to support broad statistical inferences. No statistical tests or power analyses were conducted because the research questions focused on identifying interaction strategies through observation and self-report, not on testing hypotheses or comparing conditions. We will include any omitted demographic details, exclusion criteria, and study procedure specifics in the revised manuscript to enhance transparency. The abstract and discussion emphasize the descriptive nature of the findings and do not assert statistical generalizability. revision: partial

  2. Referee: [Results / Analysis] Results / Analysis: The quantitative usage data and questionnaire feedback are presented without baseline comparisons, effect sizes, or inferential statistics, so the claim that LLM-assisted systems 'can be used productively' is supported only by existence (successful sessions occurred) rather than by evidence that the observed patterns exceed chance or prior non-LLM interfaces.

    Authors: The productivity claim is framed as a feasibility demonstration: all participants were able to complete the 3D scene manipulation tasks using the LLM-assisted system within the immersive environment. This is evidenced by the usage logs showing successful interactions and positive questionnaire feedback on usability. As the first study of its kind focusing on LLM integration in this context, we did not include baseline conditions or non-LLM comparisons, which would be valuable for future comparative work but beyond the scope of identifying LLM-specific patterns. The mixed-methods data provide rich descriptive insights into strategies and barriers, supporting the design recommendations. We can clarify the wording in the abstract and discussion to emphasize the observational basis of the findings. revision: no

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is an empirical user study reporting observational findings from 12 participants on LLM-assisted 3D scene manipulation. It contains no equations, derivations, fitted parameters, predictions, or modeling steps. The central claim is a feasibility demonstration supported directly by the collected usage data and questionnaire results. No load-bearing steps reduce to self-definition, self-citation chains, or renamed inputs. This is a standard non-circular empirical report.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical user study; it introduces no free parameters, mathematical axioms, or invented entities.

pith-pipeline@v0.9.0 · 5648 in / 1091 out tokens · 22763 ms · 2026-05-23T18:39:59.422805+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · 4 internal anchors

  1. [1]

    GPT-4 Technical Report

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Aghel Manesh, T

    S. Aghel Manesh, T. Zhang, Y . Onishi, K. Hara, S. Bateman, J. Li, and A. Tang. How people prompt generative ai to create interactive vr scenes. In Proceedings of the 2024 ACM Designing Interactive Systems Conference, pp. 2319–2340, 2024. 2

    work page 2024

  3. [3]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  4. [4]

    Beyan, A

    C. Beyan, A. Vinciarelli, and A. D. Bue. Co-located human–human interaction analysis using nonverbal cues: A survey. ACM Computing Surveys, 56(5):1–41, 2023. 2

    work page 2023

  5. [5]

    Bozkir, S

    E. Bozkir, S. ¨Ozdel, K. H. C. Lau, M. Wang, H. Gao, and E. Kas- neci. Embedding large language models into extended reality: Op- portunities and challenges for inclusion, engagement, and privacy. In Proceedings of the 6th ACM Conference on Conversational User In- terfaces, pp. 1–7, 2024. 2

    work page 2024

  6. [6]

    Brooke et al

    J. Brooke et al. Sus-a quick and dirty usability scale. Usability evalu- ation in industry, 189(194):4–7, 1996. 4

    work page 1996

  7. [7]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amod...

    work page 1901

  8. [8]

    De La Torre, C

    F. De La Torre, C. M. Fang, H. Huang, A. Banburski-Fahey, J. Amores Fernandez, and J. Lanier. Llmr: Real-time prompting of interactive worlds using large language models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–22,

    work page

  9. [9]

    Dudley, H

    J. Dudley, H. Benko, D. Wigdor, and P. O. Kristensson. Performance envelopes of virtual keyboard text input strategies in virtual reality. In 2019 IEEE International Symposium on Mixed and Augmented Real- ity (ISMAR), pp. 289–300. IEEE, 2019. 2

    work page 2019

  10. [10]

    A. M. Feit, D. Weir, and A. Oulasvirta. How we type: Movement strategies and performance in everyday typing. In Proceedings of the 2016 chi conference on human factors in computing systems , pp. 4262–4273, 2016. 2

    work page 2016

  11. [11]

    C. R. Foy, J. J. Dudley, A. Gupta, H. Benko, and P. O. Kristensson. Understanding, detecting and mitigating the effects of coactivations in ten-finger mid-air typing in virtual reality. In Proceedings of the 2021 CHI conference on human factors in computing systems , pp. 1–11,

    work page 2021

  12. [12]

    Giunchi, N

    D. Giunchi, N. Numan, E. Gatti, and A. Steed. Dreamcodevr: Towards democratizing behavior design in virtual reality with speech-driven programming. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pp. 579–589. IEEE, 2024. 1, 2, 9

    work page 2024

  13. [13]

    J. Guo, V . Mohanty, J. H. Piazentin Ono, H. Hao, L. Gou, and L. Ren. Investigating interaction modes and user agency in human-llm collab- oration for domain-specific data analysis. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems , pp. 1–9,

    work page

  14. [14]

    S. G. Hart and L. E. Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, vol. 52, pp. 139–183. Elsevier, 1988. 4

    work page 1988

  15. [15]

    K. He, A. Lapham, and Z. Li. Enhancing narratives with saymotion’s text-to-3d animation and llms. In ACM SIGGRAPH 2024 Real-Time Live!, pp. 1–2. 2024. 2

    work page 2024

  16. [16]

    Huang, P

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning , pp. 9118–

    work page

  17. [17]

    Inner Monologue: Embodied Reasoning through Planning with Language Models

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al. Inner monologue: Em- bodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  18. [18]

    Jebeli, L

    A. Jebeli, L. K. Chen, K. Guerrerio, S. Papparotto, L. Berlin, and B. J. Harden. Quantifying the quality of parent-child interaction through machine-learning based audio and video analysis: Towards a vision of ai-assisted coaching support for social workers. ACM Journal on Computing and Sustainable Societies, 2(1):1–21, 2024. 2

    work page 2024

  19. [19]

    Jiang, M

    L. Jiang, M. Phutane, and S. Azenkot. Beyond audio description: Exploring 360 video accessibility with blind and low vision users through collaborative creation. In Proceedings of the 25th interna- tional ACM SIGACCESS conference on computers and accessibility , pp. 1–17, 2023. 2

    work page 2023

  20. [20]

    Konenkov, A

    M. Konenkov, A. Lykov, D. Trinitatova, and D. Tsetserukou. Vr- gpt: Visual language model for intelligent virtual reality applications. arXiv preprint arXiv:2405.11537, 2024. 2

    work page arXiv 2024

  21. [21]

    Kurai, T

    R. Kurai, T. Hiraki, Y . Hiroi, Y . Hirao, M. Perusquia-Hernandez, H. Uchiyama, and K. Kiyokawa. Magicitem: Dynamic behavior de- sign of virtual objects with large language models in a consumer meta- verse platform. arXiv preprint arXiv:2406.13242, 2024. 1, 9

    work page arXiv 2024

  22. [22]

    J. Lee, J. Wang, E. Brown, L. Chu, S. S. Rodriguez, and J. E. Froehlich. Gazepointar: A context-aware multimodal voice assis- tant for pronoun disambiguation in wearable augmented reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–20, 2024. 2

    work page 2024

  23. [23]

    Q. V . Liao and J. W. Vaughan. AI transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.01941, pp. 5368–5393, 2023. 1, 9

    work page arXiv 2023

  24. [24]

    X. Ma, Y . Bhalgat, B. Smart, S. Chen, X. Li, J. Ding, J. Gu, D. Z. Chen, S. Peng, J.-W. Bian, et al. When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:2405.10255, 2024. 2

    work page arXiv 2024

  25. [25]

    Manfredi, U

    G. Manfredi, U. Erra, and G. Gilio. A mixed reality approach for in- novative pair programming education with a conversational ai virtual avatar. In Proceedings of the 27th International Conference on Eval- uation and Assessment in Software Engineering , pp. 450–454, 2023. 2

    work page 2023

  26. [26]

    Hello GPT-4o, May 2024

    OpenAI. Hello GPT-4o, May 2024. Available at https://openai. com/index/hello-gpt-4o . 2

    work page 2024

  27. [27]

    Plopski, T

    A. Plopski, T. Hirzle, N. Norouzi, L. Qian, G. Bruder, and T. Lan- glotz. The eye in extended reality: A survey on gaze interaction and eye tracking in head-worn extended reality. ACM Computing Surveys (CSUR), 55(3):1–39, 2022. 2

    work page 2022

  28. [28]

    Rabsahl, T

    S. Rabsahl, T. Satzger, S. Kalamkar, J. Grubert, and F. Beck. Sym- bolic event visualization for analyzing user input and behavior of aug- mented reality sessions. 2023. 2

    work page 2023

  29. [29]

    Rakkolainen, A

    I. Rakkolainen, A. Farooq, J. Kangas, J. Hakulinen, J. Rantala, M. Tu- runen, and R. Raisamo. Technologies for multimodal interaction in extended reality—a scoping review. Multimodal Technologies and In- teraction, 5(12):81, 2021. 1

    work page 2021

  30. [30]

    Roberts, A

    J. Roberts, A. Banburski-Fahey, and J. Lanier. Steps towards prompt- based creation of virtual worlds. arXiv preprint arXiv:2211.05875 ,

    work page arXiv

  31. [31]

    Rodriguez, B

    R. Rodriguez, B. T. Sullivan, M. D. Barrera Machuca, A. U. Batmaz, C. Tornatzky, and F. R. Ortega. An artists’ perspectives on natural interactions for virtual reality 3d sketching. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–20,

    work page

  32. [32]

    Scholz, T

    F. Scholz, T. E. Kolb, and J. Neidhardt. Classifying user roles in online news forums: A model for user interaction and behavior analysis. In Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization, pp. 240–249, 2024. 2

    work page 2024

  33. [33]

    Schrepp, A

    M. Schrepp, A. Hinderks, and J. Thomaschewski. Design and evalu- ation of a short version of the user experience questionnaire (ueq-s). International Journal of Interactive Multimedia and Artificial Intelli- gence, 4 (6), 103-108., 2017. 4, 5

    work page 2017

  34. [34]

    J. Song, B. Wang, Z. Wang, and D. K.-M. Yip. From expanded cinema to extended reality: How ai can expand and extend cinematic experi- ences. In Proceedings of the 16th International Symposium on Visual Information Communication and Interaction, pp. 1–5, 2023. 2

    work page 2023

  35. [35]

    Spittle, M

    B. Spittle, M. Frutos-Pascual, C. Creed, and I. Williams. A review of interaction techniques for immersive environments. IEEE Trans- actions on Visualization and Computer Graphics , 29(9):3900–3921,

    work page

  36. [36]

    J. R. Trippas, S. F. D. Al Lawati, J. Mackenzie, and L. Gallagher. What do users really ask large language models? an initial log analysis of google bard interactions in the wild. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2703–2707, 2024. 2

    work page 2024

  37. [37]

    Tsimpoukelli, J

    M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill. Multimodal few-shot learning with frozen language mod- els. Advances in Neural Information Processing Systems, 34:200–212,

    work page

  38. [38]

    A. S. Williams and F. R. Ortega. Understanding gesture and speech multimodal interactions for manipulation tasks in augmented reality using unconstrained elicitation. Proceedings of the ACM on Human- Computer Interaction, 4(ISS):1–21, 2020. 2

    work page 2020

  39. [39]

    P. C. Wright, R. E. Fields, and M. D. Harrison. Analyzing human- computer interaction as distributed cognition: the resources model. Human-Computer Interaction, 15(1):1–41, 2000. 2

    work page 2000

  40. [40]

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 2, 9

    work page arXiv 2023

  41. [41]

    Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models

    Y . Zhang, Y . Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y . Zhang, Y . Chen, et al. Siren’s song in the ai ocean: a sur- vey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. 9

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  42. [42]

    X. Zhou, A. S. Williams, and F. R. Ortega. Eliciting multimodal ges- ture+ speech interactions in a multi-object augmented reality environ- ment. In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology, pp. 1–10, 2022. 2

    work page 2022

  43. [43]

    Zimmerer, E

    C. Zimmerer, E. Wolf, S. Wolf, M. Fischbach, J.-L. Lugrin, and M. E. Latoschik. Finally on par?! multimodal and unimodal interaction for open creative design tasks in virtual reality. In Proceedings of the 2020 international conference on multimodal interaction , pp. 222– 231, 2020. 2

    work page 2020