pith. sign in

arxiv: 2606.02425 · v1 · pith:HQWMR7RPnew · submitted 2026-06-01 · 💻 cs.HC · cs.MM

Fostering Emotional Perspective-Taking: An Exploration of Affective Face-Tracking Interactions in the VR Narrative Rekindle

Hector Fan , Casper Hartveld , Mark Sivak This is my paper

Pith reviewed 2026-06-28 12:41 UTC · model grok-4.3

classification 💻 cs.HC cs.MM
keywords affective interactionemotional perspective-takingVR narrativeface-trackinginteractive digital narrativeembodied characterRekindlebiometric data
0
0 comments X

The pith

VR face-tracking recognizes player emotions to foster perspective-taking with story characters in narrative experiences.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes an experimental model for interactive digital narratives in VR that uses the headset's face-tracking to detect the player's emotional states in real time. This detection is used to create interactions that encourage emotional perspective-taking, where the player connects more closely with the emotions of their embodied character. Unlike previous approaches that use emotion data only for adjusting difficulty or visuals, this model integrates it directly into the narrative mechanics. A sympathetic reader would care if this leads to VR stories that feel more personally engaging and emotionally resonant through biometric feedback.

Core claim

The paper claims that by using a VR headset's built-in face-tracking to recognize player emotional states and translate them into narrative elements, an affective interaction model can foster emotional perspective-taking between the player and their story character, deepening emotional connection and narrative engagement in the specific VR experience Rekindle.

What carries the argument

The affective interaction model that interprets face-tracking data as emotional states to influence narrative progression and character embodiment for emotional perspective-taking.

If this is right

  • The model influences the core narrative experience rather than just system parameters like difficulty.
  • Focus on a specific authored narrative allows tailored emotional mechanics.
  • Player emotional states can be leveraged to increase connection to the embodied character.
  • Real-time biometric data from consumer VR can be integrated into IDN design.
  • Existing superficial uses of emotion input are insufficient for deep engagement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This approach might be tested in other VR narratives beyond Rekindle to see if the effect generalizes.
  • Combining face-tracking with other biometrics like eye-tracking could enhance the model further.
  • Designers could create stories where the character's reactions mirror detected player emotions to build empathy.
  • The method raises questions about privacy in using personal biometric data in entertainment.

Load-bearing premise

Face-tracking data from VR headsets can be accurately interpreted as specific emotional states and effectively translated into narrative mechanics that produce measurable increases in perspective-taking and engagement.

What would settle it

An experiment comparing the affective model to a non-emotion-tracking version of the same narrative showing no significant difference in measures of emotional connection or narrative engagement.

Figures

Figures reproduced from arXiv: 2606.02425 by Casper Hartveld, Hector Fan, Mark Sivak.

Figure 1
Figure 1. Figure 1: Affective emotional perspective-taking interaction in Rekindle. Left: a within-experience UI reflects the player’s [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The ten memory fragments scattered throughout Rekindle’s virtual environment, presented as glowing particles of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The face-tracking calibration initiates at the begin [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of the original memory retrieval mechanic (left), and the affective interaction model (right), which [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Proposed between-subjects study procedure comparing the original Rekindle with the affective version. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
read the original abstract

Interest in leveraging emotions in Interactive Digital Narrative (IDN) has been growing, and Virtual Reality (VR) offers rich access to real-time biometric data such as facial expressions; yet this capability remains underexplored in novel IDN design. Existing approaches typically treat emotion input superficially, such as adjusting system difficulty or aesthetics, but rarely influence how players experience the narrative itself. Prior work also lacks a focus on a specific authored narrative. We propose an experimental affective interaction model that uses a VR headset's built-in face-tracking capability to recognize player emotional states, fostering "emotional perspective-taking" between the player and their embodied story character, thereby deepening the player's emotional connection to the character and their narrative engagement with the VR experience.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript proposes an experimental affective interaction model for the VR narrative 'Rekindle' that uses a VR headset's built-in face-tracking to recognize player emotional states. The model is intended to foster 'emotional perspective-taking' between the player and their embodied story character, thereby deepening emotional connection and narrative engagement. The work is framed as an exploration addressing gaps in Interactive Digital Narrative (IDN) design, where prior approaches treat emotion input superficially rather than integrating it into the narrative itself.

Significance. If implemented and shown to function as described, the proposal could advance IDN research by demonstrating a deeper integration of consumer-grade biometric data into authored narratives, moving beyond adjustments to difficulty or aesthetics. This has potential to open new avenues for emotional engagement in VR storytelling, provided the mapping from face-tracking to narrative mechanics can be made concrete and testable.

major comments (1)
  1. [Abstract] Abstract: The central claim that the model fosters emotional perspective-taking and deepens narrative engagement rests on the unelaborated assumption that face-tracking data can be reliably mapped to specific emotional states and then translated into narrative mechanics; no details on the recognition algorithm, state-to-mechanics mapping, or concrete narrative examples from 'Rekindle' are supplied, leaving the feasibility of the proposal unassessable.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address the major comment point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that the model fosters emotional perspective-taking and deepens narrative engagement rests on the unelaborated assumption that face-tracking data can be reliably mapped to specific emotional states and then translated into narrative mechanics; no details on the recognition algorithm, state-to-mechanics mapping, or concrete narrative examples from 'Rekindle' are supplied, leaving the feasibility of the proposal unassessable.

    Authors: We agree that the abstract does not sufficiently preview the technical and narrative details elaborated in Sections 3 and 4 of the manuscript. The work is framed as a design exploration of a proposed affective interaction model rather than a validated implementation. In the revised version we will expand the abstract to include: (1) a high-level description of the recognition approach using the headset's built-in facial action unit tracking mapped to valence-arousal dimensions via established libraries; (2) one or two concrete examples of state-to-mechanics mappings drawn from the Rekindle narrative (e.g., detected player sadness modulating the embodied character's empathetic responses in key dialogue branches); and (3) an explicit statement that the model is offered as a testable design framework. These additions will make the proposal's feasibility more readily assessable while preserving its exploratory character. revision: yes

Circularity Check

0 steps flagged

No significant circularity; design proposal with no derivations or self-referential fits

full rationale

The paper is an explicit design proposal for an affective VR interaction model rather than a quantitative derivation or empirical claim. It contains no equations, fitted parameters, or load-bearing self-citations that reduce any result to its own inputs by construction. The central proposal identifies reliability of face-tracking and narrative impact as open questions to be addressed, not premises treated as established. This matches the default expectation of no circularity for non-derivational work.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is a design proposal with no mathematical model, so it introduces no free parameters, axioms, or invented entities beyond the high-level concept of emotional perspective-taking.

pith-pipeline@v0.9.1-grok · 5657 in / 1114 out tokens · 22097 ms · 2026-06-28T12:41:28.172847+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

  1. [1]

    Fred Charles, Samuel Lemercier, Thurid Vogt, Nikolaus Bee, Maurizio Mancini, Jérôme Urbain, Marc Price, Elisabeth André, Catherine Pélachaud, and Marc Cavazza. 2007. Affective Interactive Narrative in the CALLAS Project. InVirtual Storytelling. Using Virtual Reality Technologies for Storytelling, Marc Cavazza and Stéphane Donikian (Eds.). Springer, Berlin...

    work page doi:10.1007/978- 2007

  2. [2]

    Martinez

    Shichuan Du, Yong Tao, and Aleix M. Martinez. 2014. Compound Facial Expres- sions of Emotion.Proceedings of the National Academy of Sciences111, 15 (2014). doi:10.1073/pnas.1322355111

    work page doi:10.1073/pnas.1322355111 2014

  3. [3]

    Paul Ekman and Wallace V. Friesen. 1978. Facial Action Coding System. (1978). doi:10.1037/t27734-000

    work page doi:10.1037/t27734-000 1978

  4. [4]

    Hector Fan and Jay David Bolter. 2024. Rekindle: Enhancing Interactive Narra- tive in Virtual Reality with Coherent, Agency-driven Interactions. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–5. doi:10.1145/3613905.3648673

    work page doi:10.1145/3613905.3648673 2024

  5. [5]

    Flying Mollusk. 2015. Nevermind

    2015

  6. [6]

    Stephen Gilroy, Julie Porteous, Fred Charles, and Marc Cavazza. 2012. Exploring Passive User Interaction for Adaptive Narratives. InProceedings of the 2012 ACM International Conference on Intelligent User Interfaces. ACM, Lisbon Portugal, 119–128. doi:10.1145/2166966.2166990

    work page doi:10.1145/2166966.2166990 2012

  7. [7]

    GoodbyeWorld Games. 2021. Before Your Eyes

    2021

  8. [8]

    Andreas Halbig and Marc Erich Latoschik. 2021. A Systematic Review of Physi- ological Measurements, Factors, Methods, and Applications in Virtual Reality. Frontiers in Virtual Reality2 (2021). doi:10.3389/frvir.2021.694567

    work page doi:10.3389/frvir.2021.694567 2021

  9. [9]

    Samory Houzangbe, Olivier Christmann, Geoffrey Gorisse, and Simon Richir. 2018. Fear as a Biofeedback Game Mechanic in Virtual Reality: Effects on Engagement and Perceived Usability. InProceedings of the 13th International Conference on the Foundations of Digital Games (FDG ’18). Association for Computing Machinery, New York, NY, USA, 1–6. doi:10.1145/323...

    work page doi:10.1145/3235765.3235787 2018

  10. [10]

    Noam Knoller. 2010. Agency and the Art of Interactive Digital Storytelling. In Interactive Storytelling, Ruth Aylett, Mei Yii Lim, Sandy Louchart, Paolo Petta, and Mark Riedl (Eds.). Springer, Berlin, Heidelberg, 264–267. doi:10.1007/978-3- 642-16638-9_38

    work page doi:10.1007/978-3- 2010

  11. [11]

    Liting Kway and Alex Mitchell. 2018. Perceived Agency as Meaningful Expression of Playable Character Personality Traits in Storygames. InInteractive Storytelling, Rebecca Rouse, Hartmut Koenitz, and Mads Haahr (Eds.). Springer International Publishing, Cham, 230–239. doi:10.1007/978-3-030-04028-4_23

    work page doi:10.1007/978-3-030-04028-4_23 2018

  12. [12]

    Stacy Marsella, Jonathan Gratch, and Jeff Rickel. 2004. Expressive Behaviors for Virtual Worlds. InLife-Like Characters, M. Gabbay, Jörg Siekmann, Hel- mut Prendinger, and Mitsuru Ishizuka (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 317–360. doi:10.1007/978-3-662-08373-4_14

    work page doi:10.1007/978-3-662-08373-4_14 2004

  13. [13]

    Random graphs and branching processes

    Stacy C. Marsella. 2003. Interactive Pedagogical Drama: Carmen’s Bright IDEAS Assessed. InIntelligent Virtual Agents, Thomas Rist, Ruth S. Aylett, Daniel Ballin, and Jeff Rickel (Eds.). Springer, Berlin, Heidelberg, 1–4. doi:10.1007/978-3-540- 39396-2_1

    work page doi:10.1007/978-3-540- 2003

  14. [14]

    Stacy C Marsella, W Lewis Johnson, and Catherine M LaBore. 2003. Interactive Pedagogical Drama for Health Interventions. In11th International Conference on Artificial Intelligence in Education, Sydney, Australia. 341–348

    2003

  15. [15]

    Meta Platforms, Inc. 2025. Face Tracking for Movement SDK for Unity. Meta Platforms, Inc

    2025

  16. [16]

    Janet H. Murray. 2017.Hamlet on the Holodeck: The Future of Narrative in Cy- berspace(updated edition ed.). MIT Press, Cambridge, MA, USA

    2017

  17. [17]

    Nice Dream. 2025. Goodnight Universe

    2025

  18. [18]

    Thorben Ortmann, Qi Wang, and Larissa Putzar. 2023. Facial Emotion Recognition in Immersive Virtual Reality: A Systematic Literature Review. InProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’23). Association for Computing Machinery, New York, NY, USA, 77–82. doi:10.1145/3594806.3594861

    work page doi:10.1145/3594806.3594861 2023

  19. [19]

    Clore, and Allan Collins

    Andrew Ortony, Gerald L. Clore, and Allan Collins. 2022.The Cognitive Structure of Emotions. Cambridge University Press, Cambridge, United Kingdom ; New York, NY

    2022

  20. [20]

    Semi Ryu, Miles Popiela, Henry Bryant, Matthew Labella, Hanna Chou, Chrystine Rayburn, and Alberto Cano. 2025. SentimentVoice: Integrating Emotion AI and VR in Performing Arts. InProceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Immersive Pavilion (SIGGRAPH Immersive Pavilion ’25). Association for Computi...

    work page doi:10.1145/3721245.3734053 2025