Interactive Holographic Visualization for 3D Facial Avatar

Dinh Tuan Tran; Fujii Yasuyuki; Joo-Ho Lee; Tri Tung Nguyen Nguyen

Interactive Holographic Visualization for 3D Facial Avatar

T0 review · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read New pipeline projects 3D holographic patient faces for pain training

desk verdict A plausible proof-of-concept integration of a facial reaction predictor and a light-field display avatar, but the central real-time and training-impact claims are unmeasured. read the letter →

arxiv 2502.08085 v1 pith:3LE4XY7M submitted 2025-02-12 cs.GR

Tri Tung Nguyen Nguyen , Fujii Yasuyuki , Dinh Tuan Tran , Joo-Ho Lee This is my paper · ORCID

classification cs.GR

keywords holographicdisplay3DGaussianSplattingfacialexpressionpredictionpainassessmentmedicaltraininglight-fieldmultimodaldeeplearningFLAME

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a proof-of-concept pipeline for medical training: it takes a conversation's audio and facial video, predicts the listener's next facial expressions with a Transformer model, and projects those expressions as a 3D holographic avatar that multiple trainees can view from different angles without headsets. The authors argue that flat screens and static mannequins are poor for learning pain recognition, and that a view-dependent holographic simulated patient would be more realistic and support group training. Their concrete claim is that combining 3D Gaussian Splatting rendering with light-field calibration makes such an avatar feasible in real time, significantly improving the training environment for accurate pain assessment. The reported evaluation, however, quantifies only the expression predictor; the holographic projection itself is described as a system design.

What carries the argument

The load-bearing mechanism is the pipeline's three-stage chaining. (1) FLAME, a parametric model of facial shape, pose, and expression, supplies the low-dimensional expression code $\psi$ that both the predictor outputs and the avatar consumes. (2) The 3D Gaussian Splatting avatar represents the head as a set of Gaussian primitives blended by FLAME blend shapes, rasterized from arbitrary viewpoints at high frame rate. (3) The calibration function $\Phi_C(fov, d, \alpha_{offset}, t_{offset}, ar)$ computes, for each of 48 discrete views, a perspective view matrix and an asymmetric projection matrix so that the views merge correctly at the viewer's position; the rendered views are packed into a light-field quilt for the display. The paper's own quantitative evaluation in Table 1 covers only step (1); steps (2) and (3) are presented as a design with algorithm pseudocode but no measured rendering or display outcomes.

What would settle it

Run a controlled study in which trainees assess pain from (a) the holographic avatar, (b) a flat-screen video showing the same predicted expressions, and (c) a static mannequin; if trainees do not recognize pain significantly more accurately with the holographic display than with the flat-screen video, the paper's claim that the holographic pipeline significantly improves the training environment is not supported.

Watch

Extended reading notes

Core claim

The paper's central discovery is a way to connect a multimodal generative model to a multiview holographic display: the Transformer-based predictor outputs FLAME expression parameters; a 3D Gaussian Splatting avatar, rigged to a FLAME mesh, turns those parameters into a photorealistic head; and a calibration function computed from the display's field of view, the viewer's distance and angle, and the aspect ratio yields 48 offset perspective-projection matrices that are stitched into a light-field 'quilt' for the display. This closes the loop from conversational stimuli to a shared, headset-free 3D view of a simulated patient's non-verbal pain signals. The paper's quantitative evidence is limited to the expression-prediction step, where its model reports higher scores on appropriateness, diversity, realism, and synchrony metrics than the listed baselines.

Load-bearing premise

The central assumption, never tested, is that the 3D rendering and holographic display faithfully show the predicted facial expressions in real time and that this presentation makes pain-recognition training better.

Editorial extensions

If this is right

If the pipeline works as claimed, pain-assessment training could move from individual VR headsets to shared, headset-free group sessions where multiple trainees see the same simulated patient from their own perspective.
The same FLAME-to-Gaussian-avatar rendering path could be reused for other clinical communication skills, such as breaking bad news or detecting signs of depression, by retraining only the expression predictor.
Because the predictor consumes audio and facial video streams, the system could in principle run live, with an instructor or teleoperated actor driving the simulated patient's reactions in real time.
The quantitative improvement in expression-prediction metrics over the listed baselines suggests the Transformer design is a workable front end, independent of whether the display hardware lives up to the holographic promise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the paper's central claim would be an end-to-end user study: train one group with the holographic avatar, another with a flat-screen video of the same predicted expressions, and a third with a static mannequin, then compare their pain-recognition accuracy; the paper stops before that comparison.
The light-field calibration could be extended beyond a single horizontal row of 48 views to full parallax, and the simplifying assumption that viewers sit at a fixed distance and angle could be relaxed with eye-tracking or depth sensing.
Adding a pain-intensity label to the latent space, which the authors mention as future work, would turn the non-verbal feedback generator into a controllable intensity dial for graded training scenarios.
The predictor's reliance on tokenized action output (argmax over tokens) may limit lip-sync smoothness; a continuous regression head is a natural alternative that could be compared on the same metrics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper is a systems integration prototype, not a validated training tool. The only hard numbers are for the facial reaction predictor; everything about the holographic display and the claimed improvement in pain-assessment training is asserted, not shown.

What it does well: the pipeline is clearly described from multimodal audio/visual input through FLAME expression codes, to a 3D Gaussian avatar, to a 48-view light-field quilt on a Looking Glass display. Algorithm 1 gives a concrete, reproducible recipe for generating the 48 perspective views with the appropriate projection offsets. The predictor is compared against REACT baselines and comes out ahead on appropriateness and diversity (FRCorr 0.52 vs. 0.31 for FSQ-Tformer; FRDiv 0.1212 vs. 0.1164). That is a legitimate, if incremental, empirical result. The citations look appropriate, and the self-citation to [1] is not a problem.

The soft spots are substantial. The evaluation in Table 1 covers only the generative predictor, not the 3DGS renderer or the display. No end-to-end latency is reported. Algorithm 1 loops over 48 views, and the only speed figure in the paper is ≥60 fps for a single 2D view on an RTX 4060. If those views are rendered sequentially, a full quilt would take roughly 800 ms per frame, which is not interactive. The paper never states that the views are batched or that the light-field shader amortizes the cost, so the "real-time interactive" claim in the abstract is not backed by any measurement. The stress-test note is on target: this is a correctness risk, not just a missing user study. There is also no user study or any qualitative evaluation of the hologram itself, so the claim that the system "significantly improves the training environment" is an overclaim. The paper's own framing as a "proof-of-concept" is more honest than the abstract, but the gap between the two is large.

Lack of code and data makes the predictor results hard to verify, though the metric table is at least partially interpretable. The related-work section's claim that "no existing solution" exists for dynamic 3D facial avatar projection on light-field displays is a bit strong, but that is a minor sin for a workshop-style paper.

Who is this for? People working on holographic avatar interfaces or medical simulation systems might want to read it as a system sketch. It does not deserve to be accepted as a validated research result in its current form, but it does deserve a serious referee if reframed as a proof-of-concept with explicit limitations and if the authors add even basic end-to-end evidence: a measured quilt frame rate, screenshots or a video of the display, and ideally a small usability observation. I would not cite it in my own work yet.

Recommendation: send it to a workshop or a short-paper track, with major revision. Ask for the missing numbers and a modest display evaluation. The core idea is reasonable; the evidence is the problem.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the predictor is trained on external dyadic data and the display calibration is a geometric algorithm, not a fit to the claimed outcome.

full rationale

The paper's derivation chain is not circular. The facial feedback predictor (Section 3.1) transforms multimodal inputs using 1D convolutions and a Transformer and is evaluated on a held-out test set against external baselines (Table 1); the paper explicitly revises the pain task into a generalized non-verbal feedback task because 'the pain-related dataset for simulated patients does not exist' and states that 'Pain intensity future may be added into the latent space,' so the pain-assessment framing is not defined by the model's own outputs. The multi-view calibration (Algorithm 1) computes 48 view matrices and projection offsets from display geometry (fov, viewing distance, alpha offset); it is a derived rendering formula, not a parameter fit to the evaluation metrics. Reference [1] is a self-citation to the authors' FSQ-Tformer, but it appears only as a baseline in Table 1 and does not justify the central pipeline. The real-time interactive claim is not circular: it is an unsupported feasibility assumption (no end-to-end 48-view frame rate is reported), which is a correctness risk, not a reduction of the claimed result to its inputs. The absence of end-to-end validation of the 3DGS avatar and Looking Glass display is a missing support, but not circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The system does not introduce new mathematical entities or physical constructs. The assumptions are about the compatibility and validity of existing components when combined, and the main missing piece is an end-to-end validation of those assumptions.

free parameters (1)

Transformer model weights of the facial feedback predictor
The predictor G is trained on dyadic interaction data, and its weights are learned parameters that determine the output expressions. The paper does not report the dataset size or training details, but the model parameters are essential to the central claim that the pipeline can generate realistic expressions.

assumptions (3)

domain assumption FLAME expression parameters from the predictor can be directly used to drive the 3D Gaussian Splatting avatar.
The paper assumes that the FLAME expression codes output by the Transformer are compatible with the GaussianAvatar's input, and that this mapping produces correct facial movements. This is not validated in the paper.
domain assumption The Looking Glass display calibration algorithm correctly generates a 48-view quilt that produces a faithful 3D hologram.
The paper describes Algorithm 1 and states it is used to project the avatar, but there is no quantitative or qualitative evaluation of the display quality, such as measuring angular resolution, ghosting, or perceived realism.
domain assumption The dyadic reaction dataset (e.g., from the REACT challenge) represents facial feedback that is also applicable to pain expressions.
The paper acknowledges that a pain-specific dataset does not exist and uses general non-verbal communication data instead. It assumes that the learned expressions will transfer to pain-related signals, which is not demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Interactive Holographic Visualization for 3D Facial Avatar." pith.science (2026). https://pith.science/paper/3LE4XY7M

@misc{pith2026250208085,
  author       = {Pith},
  title        = {Pith review of: Interactive Holographic Visualization for 3D Facial Avatar},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LE4XY7M}},
  note         = {Machine review of arXiv:2502.08085}
}

read the original abstract

Traditional methods for visualizing dynamic human expressions, particularly in medical training, often rely on flat-screen displays or static mannequins, which have proven inefficient for realistic simulation. In response, we propose a platform that leverages a 3D interactive facial avatar capable of displaying non-verbal feedback, including pain signals. This avatar is projected onto a stereoscopic, view-dependent 3D display, offering a more immersive and realistic simulated patient experience for pain assessment practice. However, there is no existing solution that dynamically predicts and projects interactive 3D facial avatars in real-time. To overcome this, we emphasize the need for a 3D display projection system that can project the facial avatar holographically, allowing users to interact with the avatar from any viewpoint. By incorporating 3D Gaussian Splatting (3DGS) and real-time view-dependent calibration, we significantly improve the training environment for accurate pain recognition and assessment.

Figures

Figures reproduced from arXiv: 2502.08085 by the authors.

**Figure 1.** 3D Holographic Avatar Projection Overview. A novel pipeline combining 3D Gaussian Splatting and light-field shading enables real-time, photo-realistic facial avatar hologram animations, streamed to the 3D Looking Glass display. The OpenGL-based light-field shading program uses FLAME expression parameters with its 3D Gaussian primitive correspondence and the display calibration parameters to render a multi-view quilt… view at source ↗

**Figure 2.** Holographic rendering pipeline overview. Our system consists of a blend-shape model for the 3D Gaus [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** 3D FLAME output features could be used to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 6 canonical work pages

[1]

Eason, B

G. Eason, B. Noble, and I. N. Sneddon, ``On certain integrals of Lipschitz-Hankel type involving products of Bessel functions,'' Phil. Trans. Roy. Soc. London, vol. A247, pp. 529--551, April 1955

1955
[2]

Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol

J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68--73
[3]

I. S. Jacobs and C. P. Bean, ``Fine particles, thin films and exchange anisotropy,'' in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271--350

1963
[4]

Elissa, ``Title of paper if known,'' unpublished

K. Elissa, ``Title of paper if known,'' unpublished
[5]

Nicole, ``Title of paper with only first word capitalized,'' J

R. Nicole, ``Title of paper with only first word capitalized,'' J. Name Stand. Abbrev., in press
[6]

Yorozu, M

Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, ``Electron spectroscopy studies on magneto-optical media and plastic substrate interface,'' IEEE Transl. J. Magn. Japan, vol. 2, pp. 740--741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982]

1987
[7]

Young, The Technical Writer's Handbook

M. Young, The Technical Writer's Handbook. Mill Valley, CA: University Science, 1989

1989
[8]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

arXiv 2021

Show all 16 references

[9]

`` F inite scalar quantization as facial tokenizer for dyadic reaction generation''

Quang Tien Dam, Tri Tung Nguyen Nguyen, Dinh Tuan Tran, and Joo-Ho Lee. `` F inite scalar quantization as facial tokenizer for dyadic reaction generation''
[10]

`` T he effect of mixed reality-based holopatient in problem-based learning contexts''

Yun Kang and Insook Lee. `` T he effect of mixed reality-based holopatient in problem-based learning contexts''. Clinical Simulation in Nursing , 82:101438, 2023

2023
[11]

Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. `` L earning a model of facial shape and expression from 4D scans''. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) , 36(6):194:1--194:17, 2017

2017
[12]

`` S tandardized patient simulation for more effective undergraduate nursing education: a systematic review and meta-analysis''

Jinkyoung Ma, Youngjin Lee, and Jiwon Kang. `` S tandardized patient simulation for more effective undergraduate nursing education: a systematic review and meta-analysis''. Clinical Simulation in Nursing , 74:19--37, 2023

2023
[13]

`` G aussianavatars: Photorealistic head avatars with rigged 3d gaussians''

Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nie ner. `` G aussianavatars: Photorealistic head avatars with rigged 3d gaussians''. arXiv preprint arXiv:2312.02069 , 2023

2023 arXiv
[14]

`` O rchestrating virtual reality simulations in undergraduate nursing education''

Mamta Shah and J Wills-Savoia. `` O rchestrating virtual reality simulations in undergraduate nursing education''. In Companion Proceedings 13th International Conference on Learning Analytics & Knowledge (LAK23) , pages 9--12, 2023

2023
[15]

`` R eact 2024: the second multiple appropriate facial reaction generation challenge''

Siyang Song, Micol Spitale, Cheng Luo, Cristina Palmero, German Barquero, Hengde Zhu, Sergio Escalera, Michel Valstar, Tobias Baur, Fabien Ringeval, et al. `` R eact 2024: the second multiple appropriate facial reaction generation challenge''. arXiv preprint arXiv:2401.05166 , 2024

2024 arXiv
[16]

and" FUNCTION bbl.etal

document texmf/0000775000000000000000000000000014753007406010707 5ustar rootroottexmf/bibtex/0000775000000000000000000000000014753007406012164 5ustar rootroottexmf/bibtex/bst/0000775000000000000000000000000014753007406012754 5ustar rootroottexmf/bibtex/bst/test.bst000066400000...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Eason, B

G. Eason, B. Noble, and I. N. Sneddon, ``On certain integrals of Lipschitz-Hankel type involving products of Bessel functions,'' Phil. Trans. Roy. Soc. London, vol. A247, pp. 529--551, April 1955

1955

[2] [2]

Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol

J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68--73

[3] [3]

I. S. Jacobs and C. P. Bean, ``Fine particles, thin films and exchange anisotropy,'' in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271--350

1963

[4] [4]

Elissa, ``Title of paper if known,'' unpublished

K. Elissa, ``Title of paper if known,'' unpublished

[5] [5]

Nicole, ``Title of paper with only first word capitalized,'' J

R. Nicole, ``Title of paper with only first word capitalized,'' J. Name Stand. Abbrev., in press

[6] [6]

Yorozu, M

Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, ``Electron spectroscopy studies on magneto-optical media and plastic substrate interface,'' IEEE Transl. J. Magn. Japan, vol. 2, pp. 740--741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982]

1987

[7] [7]

Young, The Technical Writer's Handbook

M. Young, The Technical Writer's Handbook. Mill Valley, CA: University Science, 1989

1989

[8] [8]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

arXiv 2021