Pith. sign in

REVIEW 3 major objections 5 minor 29 references

LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM-generated JSON data, not code, can drive interactive XR worlds with fewer crashes and lower cost.

desk verdict A real systems contribution with a sensible JSON-based design, but the efficiency numbers are overstated and need a proper comparative study. read the letter →

arxiv 2502.02441 v1 pith:CYVJC2WO submitted 2025-02-04 cs.MM cs.AI

Jiangong Chen , Xiaoyi Wu , Tian Lan , Bin Li This is my paper · ORCID
classification cs.MMcs.AI
keywords ExtendedrealityLargelanguagemodelsJSONdatagenerationHuman-computerinteractionVirtualobjectcreationAnimationContext-awarepromptingUserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMER tries to establish that interactive extended-reality worlds can be built more reliably and cheaply by having a large language model output structured JSON data rather than executable code. The paper reports that a Context Library plus a two-stage prompt wrapper cuts per-request input tokens to about 3,200, an over 80% reduction against the open-source baseline, and that comparable tasks finish about 60% faster than the measured baseline tasks. Because the JSON is schema-constrained and executed by pre-built modules, the LLM never has to produce valid programs, removing runtime compilation errors and application crashes by construction. An eight-participant user study across seven XR tasks is offered as evidence that the approach is practical, with the paper itself noting that the comparison requests are similar but not identical to baseline tasks.

What carries the argument

The load-bearing object is the schema-constrained JSON output together with the multi-stage LLM Wrapper: stage one asks the LLM to classify the request and name the contextual categories it needs, and stage two pulls only that context from the Context Library and requests a final JSON response conforming to a per-task schema. Each execution module (Virtual Object Creator, Animation Library, Reality Fusion Engine) converts that JSON into XR behavior through pre-built Unity scripts, with optional collision-checking and physics post-processing to fix floating or misplaced objects. The constrained JSON subspace replaces free-form code generation, which is what removes compile-time failures and shortens response latency.

What would settle it

Run the same set of user requests and tasks through LLMER, the open-source baseline, and the speech-driven VR baseline under identical hardware and network conditions; if the per-request token count and task-completion-time gaps become much smaller than reported or disappear, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

The central claim is that translating a natural-language request into schema-constrained JSON data, instead of into code, turns LLM-based XR generation from an open-ended code-writing problem into a constrained data-filling problem. The LLM first classifies the request and selects the contextual categories it needs; the Context Library supplies exactly that scene, object, animation, and user-state information; and the LLM fills in a JSON schema that one of three modules executes in Unity. This design eliminates the need for the LLM to generate syntactically valid scripts, so compile-time failures and runtime crashes are avoided, while token consumption stays low because each prompt is stripped of irrelevant context. The reported user study claims over 80% fewer consumed tokens per request and around 60% lower task completion time than the compared baselines, with high fulfillment ratings across the seven designed tasks.

Load-bearing premise

The efficiency result rests on the assumption that the requests and tasks compared with the baseline systems are similar enough in difficulty that the 80% token and 60% time reductions come from the JSON design rather than from easier tasks.

Editorial extensions

If this is right

  • XR scene creation no longer requires the LLM to emit valid executable code, removing a whole class of runtime compilation crashes by construction.
  • Per-request token consumption around 3,300 makes pay-per-use LLM APIs affordable enough for real-time, multi-turn voice interaction in a headset.
  • Response generation times around 10 seconds, together with high task fulfillment, indicate that voice-controlled scene editing is practical on commercial XR hardware.
  • Because new capabilities are added as animation units or module scripts rather than model retraining, the system's behavior set can grow incrementally without fine-tuning.
  • The split between constrained JSON generation and pre-built execution suggests that reliability can be improved by tightening the output format rather than by adding more code-generation iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported efficiency gains are sensitive to task-difficulty assumptions; a matched task set run on LLMER and the two baselines under identical conditions would directly test whether the savings come from JSON generation or from easier tasks.
  • The Context Library's fixed category schema, rather than the LLM itself, may be the ceiling on scene understanding; open-ended context slots could reduce the placement errors seen in Task 2 of the user study.
  • The same 'generate constrained data, then execute with pre-built modules' pattern could transfer to other embodied LLM settings, such as robot instruction, game NPC control, or smart-environment automation.
  • A hybrid that uses JSON for common operations and falls back to code for novel long-tail requests is a natural next step, and the paper's own limitation discussion points in that direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents LLMER, a system for creating interactive XR worlds by having a large language model (GPT-4) generate structured JSON data rather than executable code. The system includes a Context Library that supplies only request-relevant scene information, an LLM Wrapper that processes user requests in two stages, and three execution modules (Virtual Object Creator, Animation Library, Reality Fusion Engine). The authors report an IRB-approved user study with eight participants performing seven XR tasks, and claim over 80% reduction in consumed tokens and around 60% reduction in task completion time compared to prior code-generation approaches such as LLMR and DreamCodeVR. The paper also includes questionnaire results and interview feedback, all reported as positive overall.

Significance. If the efficiency claims were properly supported, the core design idea---constraining LLM output to a small JSON schema space and using pre-defined execution modules---would be a meaningful contribution to LLM-driven XR interaction, with potential benefits in cost, latency, and robustness. The open-source implementation, the concrete module design, and the preliminary user study are credible strengths. However, the headline quantitative claims currently rest on non-identical task comparisons, per-request rather than per-task token metrics, and a definitional zero-error-rate statement. These issues need to be addressed before the claims can be accepted as established, though they do not undermine the basic feasibility of the JSON-based approach.

major comments (3)
  1. [Section 7.1] The claimed 'over 80% reduction in consumed tokens' compares LLMER's average 3,300 tokens per request against roughly 30,000 tokens per request for LLMR, but the authors explicitly note that users may submit multiple requests per task and that they present 'single-request metrics here.' Pay-per-use API cost is determined by total tokens across all requests in a task, not per-request tokens. The paper therefore does not establish the advertised task-level token savings; please report per-task token totals for LLMER and an equivalent baseline, or clearly rephrase the claim as being per-request only.
  2. [Section 7.2] The 'around 60% reduction in task completion time' is computed against tasks that the authors concede are similar but not identical to those in DreamCodeVR. Numerically, comparing Task 3 (69.37s) with DreamCodeVR Task 1 (167.7s) gives a 58.6% reduction, while Task 4 (157.2s) versus DreamCodeVR Task 3 (271.4s) gives only a 42.1% reduction, so the 'about 60% in both tasks' statement is inaccurate for the second comparison. Moreover, no confidence intervals, standard deviations of the difference, or significance tests are provided, and the baseline values are taken from previously published work rather than measured in the same environment. Please either run the baseline systems under comparable conditions or soften the claims to descriptive observations with explicit caveats.
  3. [Sections 7.1 and 4.5] The zero-error-rate statement is circular: the paper defines error rate as runtime compilation errors (as in LLMR and DreamCodeVR), and LLMER never compiles code, so a zero rate is guaranteed by construction and does not measure generation quality or crash likelihood. Section 4.5 states that invalid JSON data leads to logged warnings and no user response, and Section 7.2 reports Task 2 placement errors (objects on the floor, in the air, or overlapping the table). The paper should report the actual rate of JSON parse failures, cases where modules fail to execute, and task fulfillment levels, and should avoid presenting zero compilation errors as equivalent to zero generation errors.
minor comments (5)
  1. [Section 7.2] There is a typo in 'implying a time reduction of about 60% in both basks'; 'basks' should be 'tasks'.
  2. [Section 7.1] The term 'request' should be defined explicitly: does it include both the initial and refined stages of the LLM Wrapper, or just one LLM API call? This matters for interpreting the per-request token counts.
  3. [Section 7.1] The sentence 'LLMER does not compile code during runtime, resulting in a zero error rate as defined in [6] and [9]' is confusing because error rate is not a property of the system alone; it should be clarified that the comparison was not measured for LLMER.
  4. [Section 4.1] The history queue capacity M is set to 10 based on observation, but no sensitivity analysis is provided; a sentence noting the choice's rationale and its potential effect on token consumption would help reproducibility.
  5. [Section 6] The sentence 'We will also analyze the insensitive statistics' should likely be 'the descriptive statistics' or 'the non-sensitive statistics'; please clarify.

Circularity Check

1 steps flagged · score 3.0 of 10

Efficiency comparisons are measured rather than derived; the only definitional circularity is the zero-error-rate claim, which counts compilation errors that JSON output cannot produce.

  1. self definitional [Section 7.1 (Cost Analysis); cf. Section 4.5 (LLM Wrapper)]
    "LLMER does not compile code during runtime, resulting in a zero error rate as defined in [6] and [9]. Thus, no error rate measurements are necessary for LLMER, as it inherently eliminates runtime compilation errors typical in similar technologies. ... Invalid JSON data will lead to a log of warnings and no response to the user's request."

    The paper presents zero error rate as evidence of reliability, but the error metric is inherited from code-generation baselines and defined over runtime compilation errors. LLMER outputs JSON consumed by pre-defined modules and never compiles code, so the compilation-error rate is identically zero by construction rather than by measurement. Parse failures and non-responses are explicitly logged as warnings rather than counted as errors, so the zero-error statement cannot support the crash-avoidance claim beyond the metric's definition. It is a tautology of the chosen error definition.

full rationale

The paper's headline quantitative claims—over 80% token reduction and around 60% task-time reduction—are direct measurements of consumed tokens and completion times, not fitted parameters or quantities derived from the system's own definitions. The baselines (LLMR and DreamCodeVR) are external works; the paper's self-citations are limited to related work and are not load-bearing. The token comparison is weakened by the authors' admission that requests 'may not be identical' and that single-request metrics are presented despite users submitting multiple requests per task, but this is a comparability limitation, not circularity. The one genuinely circular step is the zero-error-rate claim in Section 7.1: error is defined as runtime compilation errors, which JSON-based execution avoids by construction, and invalid JSON is treated as warnings rather than errors. This definitional circularity does not invalidate the measured efficiency results, so the overall score is moderate rather than high.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central efficiency claims rest on hand-selected parameters (M=10) and unverified domain assumptions about context coverage and animation coverage. No new physical or mathematical entities are introduced; the only invented components are software design choices.

free parameters (1)
  • History queue capacity M = 10
    Section 4.1 sets M=10 based on 'observation that users rarely refer to interactions occurring more than 10 rounds in the past', a hand-picked value without ablation or sensitivity analysis.
assumptions (4)
  • domain assumption The four context categories (resources, scene data, animations, user context) and their properties are sufficient to represent all relevant XR scene information for a user request.
    Section 4.1 states these categories were chosen to cover environment creation, existing scene data, ongoing animations, and user contexts, but no evidence shows they capture all cases; Task 2 placement errors suggest real-world context remains incomplete.
  • domain assumption The predefined animation units (Translate, Rotate, Gaze, Orbit, Scaling, Coloring, Attach, Detach, Catch, Stop, Destroy) cover the common animations users request in XR.
    Section 4.3 claims 'through extensive testing, we found that the existing animation units adequately cover the most commonly used animations', but no test data or coverage analysis is provided.
  • ad hoc to paper The efficiency comparison to LLMR and DreamCodeVR is valid despite non-identical tasks.
    Section 7.1 says 'the requests compared may not be identical, they are similar enough at a high level', which is an unverified assumption underlying the 80% token and 60% time reductions.
  • domain assumption GPT-4 will generate valid JSON conforming to the provided schema for the constructed prompts.
    Section 4.5 says invalid JSON is 'pretty rare throughout our testing' but no error rate or examples are given; the zero-error-rate claim is instead defined by excluding runtime compilation, which JSON avoids by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models." pith.science (2026). https://pith.science/paper/CYVJC2WO

@misc{pith2026250202441,
  author       = {Pith},
  title        = {Pith review of: LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYVJC2WO}},
  note         = {Machine review of arXiv:2502.02441}
}
read the original abstract

The integration of Large Language Models (LLMs) like GPT-4 with Extended Reality (XR) technologies offers the potential to build truly immersive XR environments that interact with human users through natural language, e.g., generating and animating 3D scenes from audio inputs. However, the complexity of XR environments makes it difficult to accurately extract relevant contextual data and scene/object parameters from an overwhelming volume of XR artifacts. It leads to not only increased costs with pay-per-use models, but also elevated levels of generation errors. Moreover, existing approaches focusing on coding script generation are often prone to generation errors, resulting in flawed or invalid scripts, application crashes, and ultimately a degraded user experience. To overcome these challenges, we introduce LLMER, a novel framework that creates interactive XR worlds using JSON data generated by LLMs. Unlike prior approaches focusing on coding script generation, LLMER translates natural language inputs into JSON data, significantly reducing the likelihood of application crashes and processing latency. It employs a multi-stage strategy to supply only the essential contextual information adapted to the user's request and features multiple modules designed for various XR tasks. Our preliminary user study reveals the effectiveness of the proposed system, with over 80% reduction in consumed tokens and around 60% reduction in task completion time compared to state-of-the-art approaches. The analysis of users' feedback also illuminates a series of directions for further optimization.

Figures

Figures reproduced from arXiv: 2502.02441 by the authors.

Figure 1
Figure 1. An illustration of our system to enhance the immersive user-XR interaction. It (i) processes language/audio inputs and (ii) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System architecture of LLMER. In the figure, blue, black, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example usage of the Context Library. Based on the JSON data [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Examples of creating complex objects using local resources. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: A request sent to the Virtual Object Creator. For requests related [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: A request sent to the Animation Library. By parsing the action [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Example applications powered by the Reality Fusion Engine. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Task completion time regarding XR familiarity. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: Number of help requests regarding XR familiarity. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Results of Questionnaire. Responses were measured on a [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 9
Figure 9. Figure 9: Fulfillment levels. Number of help requests. We also record the number of help requests by users when executing each task, as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 16 canonical work pages

  1. [1]

    Abdelnabi, A

    S. Abdelnabi, A. Gomaa, S. Sivaprasad, L. Schönherr, and M. Fritz. Llm- deliberation: Evaluating llms with interactive multi-agent negotiation games. arXiv preprint arXiv:2309.17234, 2023. 2

    arXiv 2023

  2. [2]

    magic number 5

    N. Bevan, C. Barnum, G. Cockton, J. Nielsen, J. Spool, and D. Wixon. The" magic number 5" is it enough for web testing? In CHI’03 extended abstracts on Human factors in computing systems, pp. 698–699, 2003. 6

    work page 2003

  3. [3]

    J. Chen, T. Lan, and B. Li. GPT-VR Nexus: Chatgpt-powered immersive virtual reality experience. In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 01–02. IEEE,

    work page 2024

  4. [4]

    Y . Chen, A. Pesaranghader, T. Sadhu, and D. H. Yi. Can we rely on llm agents to draft long-horizon plans? let’s take travelplanner as an example. arXiv preprint arXiv:2408.06318, 2024. 2

    arXiv 2024

  5. [5]

    S. H.-W. Chuah. Why and who will adopt extended reality technology? literature review, synthesis, and future research agenda. Literature Review, Synthesis, and Future Research Agenda (December 13, 2018), 2018. 1

    work page 2018

  6. [6]

    LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

    F. De La Torre, C. M. Fang, H. Huang, A. Banburski-Fahey, J. A. Fernan- dez, and J. Lanier. Llmr: Real-time prompting of interactive worlds using large language models. arXiv preprint arXiv:2309.12276, 2023. 1, 2, 3, 4, 7, 9

    work page Pith review arXiv 2023

  7. [7]

    Driess, F

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pp. 8469–8488. PMLR, 2023. 2

    work page 2023

  8. [8]

    C. M. Fang, K. Zieli ´nski, P. Maes, J. Paradiso, B. Blumberg, and M. B. Kjærgaard. Enabling waypoint generation for collaborative robots using llms and mixed reality. arXiv preprint arXiv:2403.09308, 2024. 2

    arXiv 2024

Show all 29 references
  1. [9]

    Giunchi, N

    D. Giunchi, N. Numan, E. Gatti, and A. Steed. Dreamcodevr: Towards de- mocratizing behavior design in virtual reality with speech-driven program- ming. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pp. 579–589. IEEE, 2024. 1, 2, 6, 7, 8, 9

    2024

  2. [10]

    R. Gong, Q. Huang, X. Ma, H. V o, Z. Durante, Y . Noda, Z. Zheng, S.-C. Zhu, D. Terzopoulos, L. Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971, 2023. 2

    2023 arXiv

  3. [11]

    Gundawar, M

    A. Gundawar, M. Verma, L. Guan, K. Valmeekam, S. Bhambri, and S. Kambhampati. Robust planning with llm-modulo framework: Case study in travel planning. arXiv preprint arXiv:2405.20625, 2024. 2

    2024 arXiv

  4. [12]

    T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024. 2

    2024 arXiv

  5. [13]

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023. 3

    2023

  6. [14]

    Huang, F

    H. Huang, F. De La Torre, C. M. Fang, A. Banburski-Fahey, J. Amores, and J. Lanier. Real-time animation generation and control on rigged models via large language models. arXiv preprint arXiv:2310.17838, 2023. 2

    2023 arXiv

  7. [15]

    Kambhampati, K

    S. Kambhampati, K. Valmeekam, L. Guan, K. Stechly, M. Verma, S. Bham- bri, L. Saldyt, and A. Murthy. Llms can’t plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817, 2024. 2

    2024 arXiv

  8. [16]

    Kurai, T

    R. Kurai, T. Hiraki, Y . Hiroi, Y . Hirao, M. Perusquia-Hernandez, H. Uchiyama, and K. Kiyokawa. Magicitem: Dynamic behavior design of virtual objects with large language models in a consumer metaverse platform. arXiv preprint arXiv:2406.13242, 2024. 2

    2024 arXiv

  9. [17]

    J. R. Lewis. Ibm computer usability satisfaction questionnaires: psy- chometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1):57–78, 1995. 7

    1995

  10. [18]

    Y . Li, H. Wen, W. Wang, X. Li, Y . Yuan, G. Liu, J. Liu, W. Xu, X. Wang, Y . Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024. 2

    2024 arXiv

  11. [19]

    C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y . Liu, and T.-Y . Lin. Magic3d: High-resolution text-to- 3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300–309, 2023. 3

    2023

  12. [20]

    A. M. Lund. Measuring usability with the use questionnaire12. Usability interface, 8(2):3–6, 2001. 7

    2001

  13. [21]

    Structured Outputs - OpenAI API

    OpenAI. Structured Outputs - OpenAI API. 6

  14. [22]

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bern- stein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1–22, 2023. 2

    2023

  15. [23]

    Poole, A

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to- 3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022. 3

    2022 arXiv

  16. [24]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervi- sion. In International Conference on Machine Learning, pp. 28492–28518. PMLR, 2023. 3

    2023

  17. [25]

    Singh, G

    I. Singh, G. Singh, and A. Modi. Pre-trained language models as prior knowledge for playing text-based games.arXiv preprint arXiv:2107.08408,

    arXiv

  18. [26]

    C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y . Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998–3009, 2023. 2

    2023

  19. [27]

    Y . Su. V oice2action: Language models as agent for efficient real-time interaction in virtual reality. arXiv preprint arXiv:2310.00092, 2023. 2

    2023 arXiv

  20. [28]

    Z. Wang, H. Huang, Y . Zhao, Z. Zhang, and Z. Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023. 3

    2023 arXiv

  21. [29]

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. 2

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.