Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

An Empirical Evaluation of AI-Powered Non-Player Characters' Perceived Realism and Performance in Virtual Reality Environments

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4 Turbo-driven NPCs in VR reach 6.9-second replies and 6.67/10 believability.

desk verdict Useful pilot latency and usability data for an LLM-driven VR interrogation simulator, but the headline believability score does not survive arithmetic or calibration scrutiny; needs revision before the numbers are cited. read the letter →

arxiv 2507.10469 v1 pith:BRGV6Y6B submitted 2025-07-14 cs.HC cs.AIcs.MM

Mikko Korkiakoski , Saeid Sheikhi , Jesper Nyman , Jussi Saariniemi , Kalle Tapio , Panos Kostakos This is my paper · ORCID
classification cs.HCcs.AIcs.MM
keywords artificialintelligencelargelanguagemodelvirtualrealitynon-playercharacterbelievabilitylatencyusabilityGPT-4Turbo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that large language models can run non-player characters in virtual reality well enough for real interaction, and that the main barrier is timing, not conversation quality. It reports a VR interrogation simulator in which a GPT-4 Turbo suspect and a GPT-4 Turbo partner respond to spoken questions, tested by 18 participants. The measured results are an average cycle latency—the total wait from the player's question to the NPC's spoken answer—of 6909 ms, a believability score of 6.67/10, and a System Usability Scale score of 79.44, with the strongest believability ratings in social relationships, behavior, and intelligence. The paper argues that this demonstrates the potential of LLM-driven NPCs while exposing latency and emotional depth as the two challenges to solve.

What carries the argument

The mechanism that carries the argument is the interrogation loop: the player's recorded audio is sent to speech-to-text, the resulting text plus the entire chat history goes to GPT-4 Turbo, the generated text is converted to speech, and the audio is played by the NPC. The NPC 'memory' is this accumulated transcript, resent in full on every turn; that design is what makes GPT latency, and therefore the user-visible cycle latency, grow over the course of the conversation. The paper's named quantity is cycle latency, defined as the total waiting time from the end of the player's question to the start of the NPC's answer.

What would settle it

A decisive check would be to recompute the believability scores from the raw questionnaire items using the original scoring rule and see whether the category ordering survives; if social relationships no longer outrank emotion or personality, the 6.67/10 result is a scaling artifact rather than a finding about NPC realism.

Watch

Extended reading notes

Core claim

The paper claims that GPT-4 Turbo-powered NPCs can carry a believable, context-aware interrogation in VR without animation or embodied action: the suspect and partner scored 6.67/10 overall on the Virtual Agent Believability Questionnaire, with social relationships (8.24), behavior (8.09), and intelligence (7.99) rated well above emotion (6.16), personality (5.74), and agency (5.34). Usability measured 79.44 on the System Usability Scale, which the paper classifies as good. The main performance finding is an average cycle latency of 6909 ms, with a minimum of 2021 ms and a maximum beyond 24 s; GPT calls contribute the largest share (mean 3113 ms) and become slower as the conversation history grows, because every turn resends the full chat log. A regression of text-to-speech latency on GPT response length gives $R^2=0.75$, and users' average latency estimate of 8.67 s broadly tracked the measured value. The paper reads these results as evidence that LLM dialogue alone can supply much of NPC believability, with latency management as the decisive engineering problem.

Load-bearing premise

The load-bearing premise is that re-scaling the believability questionnaire's original scores to a 0-to-10 range preserves what the questionnaire measures, since the paper does not validate that transformation.

Editorial extensions

If this is right

  • Average cycle latency of 6909 ms with a 2021 ms minimum means the system can feel conversational in short exchanges but becomes noticeably slow as the interrogation proceeds.
  • Believability scores in the 8 range for social relationships, behavior, and intelligence indicate that speech and cognitive ability alone can make an NPC seem human-like, even when it is stationary and lacks agency.
  • Because GPT latency dominates the cycle and rises with chat history, the clearest path to real-time NPC dialogue is reducing the size of the context sent to the model or compressing conversation memory.
  • The SUS score of 79.44 places the system in the 'good' range, suggesting that speech-driven VR interaction is learnable even for participants with little VR experience.
  • Users' average latency estimate of 8.67 seconds sits close to the measured 6.9 seconds, suggesting that perceived waiting time tracks actual system latency within a few seconds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not run: replace the full chat-history resend with a rolling summary or retrieval of recent turns, and test whether cycle latency stops growing while believability holds; the paper's own latency model predicts that would be the main improvement.
  • The high partner-helpfulness score hints that this architecture could serve as an AI briefing assistant in serious VR applications, but the paper only sketches that possibility and does not claim to validate it.
  • The unvalidated re-scaling of the believability questionnaire suggests that future studies should pre-register their scoring transformation; otherwise, the 0-10 category scores cannot be compared across studies.
  • Because the NPCs were stationary and all actions were speech, the believability results say little about embodied realism; an equally plausible extension is that adding even simple gestures would raise the low agency and emotion scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a VR interrogation simulator in which two non-player characters (a suspect and a partner) are driven by GPT-4 Turbo, and reports a user study with 18 participants. The study measures system performance via STT, TTS, GPT, and overall cycle latency; perceived believability via a Virtual Agent Believability Questionnaire; usability via the System Usability Scale; social presence via the GEQ social presence module; and additional Likert-scale items on latency impact and partner helpfulness. The central claims are that the average cycle latency is 6.9 s, the NPCs achieve a believability score of 6.67/10, the system SUS score is 79.44, and that users 'fairly accurately estimated' the latency once a 3.0 s error margin is applied. The paper concludes that LLM-driven NPCs can provide usable and fairly believable interactions in VR, while noting that latency grows as chat history accumulates and that emotional/personality dimensions score lower.

Significance. If the reported numbers are reliable, the paper provides a useful empirical data point on the feasibility of LLM-powered NPCs in VR: the latency decomposition into STT, TTS, and GPT components is concrete, the SUS administration follows a standard instrument, and the interrogation scenario is a realistic testbed. The paper also explicitly acknowledges in Section 6.6 that the small sample size prevents concrete conclusions, which is an honest limitation. However, the believability results, which are a headline contribution, rest on an undisclosed re-scaling of the questionnaire and contain an internal arithmetic inconsistency that makes the 6.67/10 value non-reproducible. The latency-accuracy conclusion depends on a post hoc tolerance threshold rather than a pre-specified criterion. These issues undermine the quantitative grounding of the abstract's central assertions, so the paper requires revision before the findings can be accepted as stated. The concrete latency measurements and the frank limitation statement are strengths, but they do not by themselves fix the believability and latency-accuracy problems.

major comments (4)
  1. [Section 5.2, Table 3] The reported total believability of 6.67/10 is not reproducible from the category means in Table 3 using the procedure stated in the text ('summing up the category scores and dividing by the number of categories'). For the 'Both groups' column, the nine category values sum to 62.01, which divided by 9 gives 6.89, not 6.67. The guilty-group category means average 6.90 (or 6.77 if 'Overall believability' is excluded), not 6.62; the innocent-group means average 6.88 (or 6.81 without 'Overall believability'), not 6.72. The reported 6.67 equals the average of 6.62 and 6.72, but those group totals themselves do not follow from the category means under the described formula. Because the abstract and conclusions lead with 6.67/10 as evidence that the NPCs are 'fairly believable,' this arithmetic inconsistency must be resolved before the believability claim can be treated as a finding.
  2. [Section 5.2] The re-scaling of the Virtual Agent Believability Questionnaire to a 0–10 scale is not described. The text states that the original scoring was 'not intuitive' and that the results were re-scaled, but no transformation formula, original raw scores, or validation of the re-scaled metric are provided. All category scores and the total believability are reported on this re-scaled 0–10 scale, and the paper uses these numbers to make comparative statements such as 'high ratings in behavior, social relationships, and intelligence.' Without knowing the exact mapping and whether the transformed values preserve interval-scale meaningfulness, the believability results are not quantitatively interpretable. The authors should report the original scale's mean scores, the precise re-scaling formula, and a justification that the transformation is appropriate.
  3. [Section 5.1, Table 2] The 'acceptable error margin' of 3.0 seconds is defined post hoc, after inspection of the data, as the sum of 2.5 seconds for 'proximity to adjacent options' and 0.5 seconds for 'user tolerance.' No independent rationale or pre-specification is provided for either component. This margin reclassifies 4 of the 10 initially 'incorrect' latency evaluations, increasing the count of 'effectively correct' evaluations from 8/18 to 12/18 and underpinning the statement that users 'fairly accurately estimated' the average latency. Because the conclusion depends entirely on this arbitrary threshold, the claim is not supported. The authors should either adopt a pre-registered tolerance or report the accuracy across a range of plausibly justifiable error margins to show that the conclusion is not an artifact of the chosen cutoff.
  4. [Section 6.2] The statement 'We did not see large differences in the believability categories between the two groups' is made without any inferential statistics or measures of dispersion (e.g., standard deviations, confidence intervals, or significance tests). With 18 participants split into groups of about 9, descriptive differences such as visual properties (6.98 vs. 5.99) and awareness (5.44 vs. 7.11) may simply reflect sampling variability. The same issue affects the paper's characterization of category scores as 'high' or 'average' in Section 6.2: no variance information is reported anywhere for the believability questionnaire or the SUS. If comparative or evaluative claims are to be made, the authors should report appropriate statistical summaries; otherwise, these comparisons should be explicitly labeled as exploratory and not used to support conclusions.
minor comments (6)
  1. [Section 5.4] There are spelling errors: 'Pyschologial involvement' should be 'Psychological involvement' in both entries.
  2. [Section 5.2] The phrase 'the number or question in the category' should be 'the number of questions in the category.'
  3. [Table 2] The column titled 'Error (s)' is not clearly defined. It appears to be the absolute difference between the user's absolute error and the correct-option absolute error, but a reader would naturally interpret 'Error' as the user's estimation error. Please clarify the column's meaning and use a more descriptive name.
  4. [Section 5.1] The sentence 'From Appendix Figure A.9. We can also observe that the GPT API exhibited a large amount and variance in outliers' is grammatically incomplete and should read, for example, 'From Appendix Figure A.9, we can also observe that the GPT API exhibited a large number of outliers and high variance in their values.'
  5. [Abstract] The abstract reports 'an average cycle latency of 7 seconds,' while Table 1 reports 6909 ms (6.9 s). Consider reporting the precise value or the rounded value consistently.
  6. [Section 5.5] The innocent-group decision accuracy percentages (44.4%, 44.4%, 11.1%) sum to 99.9% due to rounding; this should be noted or the percentages adjusted (e.g., 44.4%, 44.5%, 11.1%).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all reported results are measured or questionnaire-based; no fitted inputs, self-citations, or imported uniqueness theorem.

full rationale

This is an empirical evaluation with no derivation chain whose conclusions are equivalent to its inputs. The central quantities are directly measured (latency timings for STT, TTS, GPT, and cycle) or collected from validated questionnaires (SUS, GEQ, Virtual Agent Believability). The believability total is an aggregation of questionnaire responses, not a prediction derived from a fitted model. The paper contains no self-citations that carry an argument, no imported uniqueness theorem, and no ansatz smuggled in via citation. The post hoc definition of a 3.0 s 'acceptable error margin' in Section 5.1 influences the claim that users estimated latency accurately, and the believability rescaling in Section 5.2 is under-disclosed with a reported total that does not reproduce from Table 3 under the stated formula; these are reproducibility and analysis-robustness concerns, not cases where an output is identical to an input by construction. No circular step can be exhibited, so the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No theoretical derivation is attempted. The quantitative claims depend on the two free parameters above, mainly the post hoc error tolerance, and on the questionnaire assumptions listed. No invented physical or conceptual entities are introduced; the chat-log 'memory' is an implementation detail.

free parameters (2)
  • Acceptable latency error margin = 3.0 seconds
    Defined post hoc in Section 5.1 as 2.5 seconds for option proximity plus 0.5 seconds for user tolerance; it reclassifies 4 of 10 initially incorrect latency evaluations as acceptable, raising the effectively correct count from 10/18 to 12/18.
  • TTS latency regression fit = slope 10.23 ms per response-length unit, intercept 581.90 ms
    Appendix Figure A.7; a descriptive linear fit relating GPT response length to TTS latency, reported with R-squared 0.75, not used for prediction outside the dataset.
assumptions (4)
  • domain assumption SUS, GEQ, and Virtual Agent Believability questionnaires are valid measures of usability and perceived realism in this VR context.
    Sections 4.2 and 5; all perceived-realism and usability conclusions rest on these self-report instruments, and the paper notes some believability items were not appropriate for the game yet were included.
  • domain assumption Likert-scale responses can be treated as interval data, averaged, and linearly re-scaled to a 0 to 10 scale.
    Section 5.2; the believability re-scaling assumes equal intervals across 7-point Likert items and that the original scoring was the only obstacle to comparability.
  • domain assumption The convenience sample of 18 university students is sufficient to support general claims about NPC believability and usability.
    Section 4.1; the authors acknowledge limited generalizability, but the conclusions in Sections 6 and 7 extrapolate beyond the sample.
  • domain assumption GPT-4 Turbo, STT, and TTS API latencies are attributed to system design rather than uncontrolled network or provider variability.
    Section 5.1; latency measurements mix local processing, network latency, and OpenAI-side queuing, with no controlled server environment or repeated trials per condition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Evaluation of AI-Powered Non-Player Characters' Perceived Realism and Performance in Virtual Reality Environments." pith.science (2026). https://pith.science/paper/BRGV6Y6B

@misc{pith2026250710469,
  author       = {Pith},
  title        = {Pith review of: An Empirical Evaluation of AI-Powered Non-Player Characters' Perceived Realism and Performance in Virtual Reality Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BRGV6Y6B}},
  note         = {Machine review of arXiv:2507.10469}
}
read the original abstract

Advancements in artificial intelligence (AI) have significantly enhanced the realism and interactivity of non-player characters (NPCs) in virtual reality (VR), creating more engaging and believable user experiences. This paper evaluates AI-driven NPCs within a VR interrogation simulator, focusing on their perceived realism, usability, and system performance. The simulator features two AI-powered NPCs, a suspect, and a partner, using GPT-4 Turbo to engage participants in a scenario to determine the suspect's guilt or innocence. A user study with 18 participants assessed the system using the System Usability Scale (SUS), Game Experience Questionnaire (GEQ), and a Virtual Agent Believability Questionnaire, alongside latency measurements for speech-to-text (STT), text-to-speech (TTS), OpenAI GPT-4 Turbo, and overall (cycle) latency. Results showed an average cycle latency of 7 seconds, influenced by the increasing conversational context. Believability scored 6.67 out of 10, with high ratings in behavior, social relationships, and intelligence but moderate scores in emotion and personality. The system achieved a SUS score of 79.44, indicating good usability. These findings demonstrate the potential of large language models to improve NPC realism and interaction in VR while highlighting challenges in reducing system latency and enhancing emotional depth. This research contributes to the development of more sophisticated AI-driven NPCs, revealing the need for performance optimization to achieve increasingly immersive virtual experiences.

Figures

Figures reproduced from arXiv: 2507.10469 by the authors.

Figure 1
Figure 1. The areas and characters featured in the simulator [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Sequence diagram of the interrogation loop At any point during the interrogation, the player has the option to consult the partner NPC for assistance. Interacting with the partner NPC follows the same process as outlined in the interrogation loop described above. The partner NPC is positioned in the observation room and has full access to the conversation logs, enabling them to hear and analyze everything discussed … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Observable Social Life Spaces: Exploring User Interpretations of agent-side life context in human-agent interaction

    cs.HC 2026-03 conditional novelty 6.0 of 10

    Seeing an AI agent's autonomous virtual life increased users' perceived equality with it in a small study, but the effect needs replication.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Lapeyrade, C

    S. Lapeyrade, C. Rey, Non-player character decision-making with prolog and ontologies, in: 2023 IEEE Conference on Games (CoG), IEEE, 2023, pp. 1–2. doi:https://doi.org/10.1109/CoG57401.2023.10333221

    arXiv 2023

  2. [2]

    Karaca, D

    Y. Karaca, D. Derias, G. Sarsar, Ai-powered procedural content gen- eration: Enhancing npc behaviour for an immersive gaming experi- ence, Available at SSRN 4663382 (2023). doi:https://dx.doi.org/ 10.2139/ssrn.4663382

    work page doi:10.2139/ssrn.4663382 2023

  3. [3]

    Filipovi´ c, The role of artificial intelligence in video game development, Kultura Polisa 20 (3) (2023) 50–67

    A. Filipovi´ c, The role of artificial intelligence in video game development, Kultura Polisa 20 (3) (2023) 50–67. URL https://www.ceeol.com/search/article-detail?id=1201751

    work page 2023

  4. [4]

    Zeng, A review of ai-based game npcs research, Applied and Com- putational Engineering 15 (2023) 155–159

    G. Zeng, A review of ai-based game npcs research, Applied and Com- putational Engineering 15 (2023) 155–159. doi:https://doi.org/10. 54254/2755-2721/15/20230827

    work page 2023

  5. [5]

    W. Xu, K. Yu, X. Meng, D. Monteiro, D. Kao, H.-N. Liang, Exploring the effect of the group size and feedback of non-player character spec- tators in virtual reality exergames, Frontiers in Psychology 14 (2023) 1079132. doi:https://doi.org/10.3389/fpsyg.2023.1079132

    work page arXiv 2023

  6. [6]

    Z. Guo, W. Xu, J. Zhang, H. Wang, C.-H. Lo, H.-N. Liang, Who’s watching me?: Exploring the impact of audience familiarity on player performance, experience, and exertion in virtual reality exergames, in: 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, 2023, pp. 622–631. doi:https://doi.org/10.1109/ ISMAR59233.2023.00077

    arXiv 2023

  7. [7]

    J. Moon, Reviews of social embodiment for design of non-player char- acters in virtual reality-based social skill training for autistic children, Multimodal Technologies and Interaction 2 (3) (2018) 53. doi:https: //doi.org/10.3390/mti2030053

    work page doi:10.3390/mti2030053 2018

  8. [8]

    T. S. Riyan, A. Pardede, F. Y. Manik, Implementation of finite state machine models on the artificial intelligence system of characters in the game” mmorpg” using rpg maker, Journal of Artificial Intelligence and Engineering Applications (JAIEA) 3 (1) (2023) 287–291. doi:https: //doi.org/10.59934/jaiea.v3i1.311. 29

    work page doi:10.59934/jaiea.v3i1.311 2023

Show all 21 references
  1. [9]

    Ribeiro de Oliveira, B

    T. Ribeiro de Oliveira, B. Biancardi Rodrigues, M. Moura da Silva, R. Antonio N. Spinass´ e, G. Giesen Ludke, M. Ruy Soares Gau- dio, G. Iglesias Rocha Gomes, L. Guio Cotini, D. da Silva Vargens, M. Queiroz Schimidt, et al., Virtual reality solutions employing artificial intel...

    2023 doi

  2. [10]

    Zheng, K

    S. Zheng, K. He, L. Yang, J. Xiong, Memoryrepository for ai npc, IEEE Access (2024). doi:https://doi.org/10.1109/ACCESS.2024. 3393485

    2024 doi

  3. [11]

    H. S. Ong, H. Junaedi, J. Santoso, Dragonfly algorithm for crowd npc movement simulation in metaverse, Bulletin of Social Informatics The- ory and Application 6 (1) (2022) 76–83. doi:https://doi.org/10. 31763/businta.v6i1.551

    2022

  4. [12]

    P. M. Strojny, N. Du˙ zma´ nska-Misiarczyk, N. Lipp, A. Strojny, Mod- erators of social facilitation effect in virtual reality: Co-presence and realism of virtual agents, Frontiers in psychology 11 (2020) 1252. doi: https://doi.org/10.3389/fpsyg.2020.01252

    2020

  5. [13]

    J.-H. Kim, J. Lee, S.-J. Kim, Navigating non-playable characters based on user trajectories with accumulation map and path similarity, Symme- try 12 (10) (2020) 1592. doi:https://doi.org/10.3390/sym12101592

    2020 doi

  6. [14]

    Breves, Reducing outgroup bias through intergroup contact with non- playable video game characters in vr, Presence 27 (3) (2020) 257–273

    P. Breves, Reducing outgroup bias through intergroup contact with non- playable video game characters in vr, Presence 27 (3) (2020) 257–273. doi:https://doi.org/10.1162/pres\_a\_00330

    2020 doi

  7. [15]

    Oumaima, L

    D. Oumaima, L. Mohamed, H. Hamid, H. Mohamed, Application of artificial intelligence in virtual reality, in: International Con- ference on Trends in Sustainable Computing and Machine Intelli- gence, Springer, 2023, pp. 67–85. doi:https://doi.org/10.1007/ 978-981-99-9436-6\_6

    2023

  8. [16]

    Pashentsev, Metaverses, artificial intelligence and challenges to psy- chological security, Politika nacionalne bezbednosti 25 (2) (2023)

    E. Pashentsev, Metaverses, artificial intelligence and challenges to psy- chological security, Politika nacionalne bezbednosti 25 (2) (2023). doi: https://doi.org/10.5937/pnb25-46760

    2023 doi

  9. [17]

    URL https://openai.com/ 30

    OpenAI, Gpt-4 turbo, large language model (2024). URL https://openai.com/ 30

    2024

  10. [18]

    Brooke, Sus: A quick and dirty usability scale, Usability evaluation in industry (1996)

    J. Brooke, Sus: A quick and dirty usability scale, Usability evaluation in industry (1996). doi:https://rickvanderzwet.nl/trac/personal/ export/104/liacs/hci/docs/SUS-questionaire.pdf

    1996

  11. [19]

    W. A. IJsselsteijn, Y. A. W. de Kort, K. Poels, The Game Ex- perience Questionnaire (2013). doi:https://research.tue.nl/en/ publications/the-game-experience-questionnaire

    2013

  12. [20]

    S. Guo, N. Adamo, C. Mousas, Developing a scale for measuring the be- lievability of virtual agents, in: J.-M. Normand, M. Sugimoto, V. Sund- stedt (Eds.), International Conference on Artificial Reality and Telexis- tence, Eurographics Symposium on Virtual Environments, Purdue...

    2023

  13. [21]

    Scikit-learn, Scikit-learn web page, URL: https://scikit-learn.org/ stable/, accessed: 16.6.2024. 31

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.