Pith. sign in

REVIEW 5 major objections 5 minor 22 references

ReactDiff: Latent Diffusion for Facial Reaction Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ReactDiff generates multiple listener facial reactions from a speaker's audio and video using a multi-modality transformer with latent diffusion, but its reported benchmark superiority conflicts with its own tables.

desk verdict The architecture is a plausible combination of known pieces, but the paper's own tables contradict its headline SOTA claims and the winning LRS variant is never defined. read the letter →

arxiv 2505.14151 v3 pith:E6OQQ45B submitted 2025-05-20 cs.CV cs.MM

Jiaming Li , Sheng Wang , Xin Wang , Yitao Zhu , Honglin Xiong , Zixu Zhuang , Qian Wang This is my paper · ORCID
classification cs.CVcs.MM
keywords reactionfacialreactdiffdiffusionwhilegenerationlatentapproaches
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReactDiff is a computer-vision model for a narrow but practical task: given a video and audio track of a person speaking to a camera, predict how the person on the other side of the conversation would move their face. In real conversations many reactions are acceptable, so the model produces several different listener reactions instead of one. The system first converts each frame into face descriptors, including valence-arousal, action units, and expression labels, and converts the audio into speech features. A transformer with cross-attention mixes the visual and acoustic streams. A diffusion process in a hidden space adds controlled randomness, so different runs produce different but context-dependent faces. The outputs are coefficients of a 3D morphable face model plus emotion values, which can be rendered into images. On the REACT2024 benchmark, the base version is reported with a correlation of 0.26 and a diversity score of 0.094, and the text claims it beats all previous methods. The same tables show PerFRDiff with correlation 0.38, higher variance diversity, and a realism score of 47.62 versus 66.72, so the headline claim is contradicted by the paper's own numbers. The version with the best table scores, ReactDiff with Listener Reaction Shifts, is mentioned in one paragraph and never defined by an equation, algorithm, or ablation. No error bars or code artifacts are available for verification. The architecture is reasonable, but the evidence as written does not support the stated superiority.
Extended reading notes

Core claim

The abstract states: 'ReactDiff significantly outperforms existing approaches, achieving a facial reaction correlation of 0.26 and diversity score of 0.094 while maintaining competitive realism.' The same claim is made for the LRS variant in Section 4.3, where ReactDiff (LRS) is said to achieve the best FRDist (79.63), FRDvs (0.1380), and FRSyn (39.40), surpassing PerFRDiff by 16%, 57%, and 13%. If correct, the joint MMT and latent-DDIM design would improve the REACT2024 facial reaction generation state of the art.

Load-bearing premise

Section 3.2 replaces raw video and audio with fixed pretrained representations f_facial and f_acoustic (Eqs. (1) and (2)): VA from ELIM, AU from ME-GraphAU, FE from ResMaskNet, and audio from wav2vec2. The whole pipeline assumes these frozen features preserve all reaction-relevant information and are aligned frame-by-frame. If any extractor is biased, noisy, or loses temporal cues, the MMT and diffusion modules cannot recover the missing information, and the benchmark gains would be an artifact of the feature extractors rather than of ReactDiff.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. ReactDiff combines a Multi-Modality Transformer (MMT) with a conditional DDIM in a latent reaction space for facial reaction generation. The pipeline extracts facial (VA, AU, FE) and acoustic (MFCC, wav2vec2) features, encodes them through intra- and inter-class attention, applies a diffusion process conditioned on the speaker's behavior constraint vector, and decodes into 3DMM and emotion coefficients. Experiments are reported on the REACT2024 offline and online challenge subsets, with the paper claiming state-of-the-art performance in appropriateness, diversity, realism, and synchrony.

Significance. If the experimental claims were correct, the combination of hierarchical multi-modal attention with latent-space diffusion would be a useful contribution to the REACT2024 challenge, which is a concrete benchmark with clear evaluation protocols. The authors also make the code open-source, and the training procedure is described in enough detail to be reimplemented. However, the central comparative evidence is internally inconsistent: the paper's own tables show that base ReactDiff is not state of the art on the metrics highlighted in the abstract, and the best reported variant (ReactDiff LRS) is never defined. The claimed significance is therefore not established by the manuscript as submitted.

major comments (5)
  1. [§4.3.1, Table 1] The text states that 'ReactDiff achieves the highest FRCorr score of 0.26 among all methods,' but Table 1 lists PerFRDiff at FRCorr 0.38, VQ-Diff at 0.30, and Liu et al. at 0.22 in the offline setting, with ReactDiff at 0.26. In the online setting, VQ-Diff at 0.30 is again above ReactDiff's 0.24. The highest-FRCorr claim is directly contradicted by the paper's own comparison table.
  2. [§4.3.2, Table 2] The text claims that 'ReactDiff demonstrates superior performance across all diversity metrics.' Table 2 shows that Liu et al. achieves FRDiv 0.1675 and PerFRDiff achieves FRDiv 0.1368, both above ReactDiff's 0.0940; PerFRDiff achieves FRVar 0.2191 versus ReactDiff's 0.0462; and Liu et al. achieves FRDvs 0.1385 versus ReactDiff's 0.1012. The only variant that reaches some top diversity scores is the undefined ReactDiff (LRS), whose mechanism appears nowhere in the method section.
  3. [§4.3.3, Table 3] For the realism metric FRRea, lower is better. Table 3 lists PerFRDiff at 47.62 and ReactDiff at 66.72, so the sentence 'ReactDiff achieves an FRRea score of 66.72, which is the best among the compared methods' is false. In the online setting, Trans-VAE at 69.19 is better than ReactDiff's 73.20, again contradicting the text that claims best realism.
  4. [§3.4, Tables 1–3] ReactDiff (LRS) is introduced in §3.4 with the single sentence that the authors 'introduce Listener Reaction Shifts (LRS) to model personalized cognitive styles,' but no architectural definition, training objective, algorithm, or hyperparameters are provided. This variant is nevertheless the one reported as best in FRDist (79.63), FRDvs (0.1380), and FRSyn (39.40). Because the variant is unspecified, the strongest reported results cannot be reproduced and cannot be attributed to the described architecture.
  5. [§4.4, Table 4] The ablation table does not support the conclusion that the diffusion component helps the headline metrics. Removing diffusion yields FRCorr 0.84 versus 0.26, FRDist 80.10 versus 86.70, and FRDvs 0.1349 versus 0.1012, all better than the full model. The text says removing diffusion 'eliminates diversity,' but FRDvs is higher without diffusion; it also says the full model has 'optimal diversity scores,' which the table does not show. The contribution of the diffusion module to the reported improvements is therefore not established.
minor comments (5)
  1. [Abstract and §4.3.1] The abstract's headline numbers (FRCorr 0.26 and diversity 0.094) are presented as superior to prior work, but Tables 1 and 2 list higher values for PerFRDiff and Liu et al.; all comparative claims in the abstract and text should be reconciled with the tables.
  2. [§3.4, Eq. (8)] The DDIM sampling schedule at test time is not specified: the paper does not state the value of eta, the number of sampling steps, or whether stochastic noise is added during generation, all of which are important for reproducing the reported diversity and realism results.
  3. [§3.2.1 and Eq. (3)] The AU feature dimension of 25,088 is unusually large and is not motivated; the paper should explain the projection of these features and why ME-GraphAU features without a classifier are used at this raw dimension.
  4. [§4.2] The training details report 'training time of two stages is 16h' but do not report inference time or per-epoch cost; these are relevant to the claimed suitability for real-time interaction scenarios.
  5. [§2.2] The citation '(Dosovitskiy, 2020)' is used to support a statement about the evolution of HCI; this reference is about vision transformers and is not appropriate here. An HCI-related reference should be used instead.
Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim is purchased with frozen feature extractors, windowed latent diffusion, and challenge-defined metrics. In addition, the LRS variant that produces the best rows is introduced without any mathematical or algorithmic specification. The paper provides no validation for these external commitments, and the best variant is therefore not independently checkable as submitted.

free parameters (5)
  • window_size = 50 frames
    Used in Section 3.3 to reshape the latent into windows of size 50. Chosen by hand with no reported sensitivity analysis.
  • diffusion_steps_T = 20
    Set in Section 3.4 for the DDIM process. No ablation on the number of steps is reported.
  • loss_weights_w1_w2 = 10 and 10
    Weights for MSE on 3DMM and L1 on emotion in Eq. (5). Chosen without a reported sweep.
  • generated_reactions_alpha = 10
    Number of reaction sequences generated per speaker sequence for evaluation, stated in the implementation details.
  • training_schedule = lr 5e-3, batch 16, 100 epochs
    Reported in Section 4.2; chosen by the authors without a reported tuning study.
assumptions (4)
  • domain assumption The frozen pretrained feature extractors faithfully encode all speaker behavior needed for listener reaction prediction.
    Section 3.2 replaces raw video and audio with features from ELIM, ME-GraphAU, ResMaskNet, and wav2vec2. The rest of the model never sees raw inputs, so any information lost by these extractors cannot be recovered.
  • domain assumption Cropping the latent into non-overlapping 50-frame windows and denoising each window independently preserves the temporal and one-to-many structure of reactions.
    Section 3.3 reshapes x0 to (N//ws, ws, 128) and Section 3.4 runs DDIM on these windows, relying on the transformer decoder to restore full-sequence temporal consistency.
  • domain assumption The REACT2024 best-match metrics, Max(CCC) and Min(DTW) over a set of appropriate ground-truth reactions, validly measure appropriateness.
    Appendix Eqs. (9) and (10) define FRCorr and FRDist using the most similar ground-truth listener reaction, so a generated sample is rewarded for matching any plausible reaction rather than the actual listener.
  • domain assumption FaceVerseV2 and PIRender can convert the predicted 3DMM and emotion coefficients into realistic images suitable for FID-based realism evaluation.
    Section 3.3 uses these renderers for visualization and realism assessment. Flaws in the renderers, such as the acknowledged face-identity bias, directly enter the FRRea metric.
invented entities (1)
  • Listener Reaction Shifts (LRS)
    purpose: Model personalized cognitive styles in generated listener reactions, inspired by PerFRDiff's weight-editing paradigm.
    The mechanism appears in one sentence in Section 3.4 and in Tables 1 to 3 as ReactDiff (LRS), but no equation, algorithm, hyperparameter, or ablation defines it. No falsifiable handle outside the paper is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReactDiff: Latent Diffusion for Facial Reaction Generation." pith.science (2026). https://pith.science/paper/E6OQQ45B

@misc{pith2026250514151,
  author       = {Pith},
  title        = {Pith review of: ReactDiff: Latent Diffusion for Facial Reaction Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6OQQ45B}},
  note         = {Machine review of arXiv:2505.14151}
}
read the original abstract

Given the audio-visual clip of the speaker, facial reaction generation aims to predict the listener's facial reactions. The challenge lies in capturing the relevance between video and audio while balancing appropriateness, realism, and diversity. While prior works have mostly focused on uni-modal inputs or simplified reaction mappings, recent approaches such as PerFRDiff have explored multi-modal inputs and the one-to-many nature of appropriate reaction mappings. In this work, we propose the Facial Reaction Diffusion (ReactDiff) framework that uniquely integrates a Multi-Modality Transformer with conditional diffusion in the latent space for enhanced reaction generation. Unlike existing methods, ReactDiff leverages intra- and inter-class attention for fine-grained multi-modal interaction, while the latent diffusion process between the encoder and decoder enables diverse yet contextually appropriate outputs. Experimental results demonstrate that ReactDiff significantly outperforms existing approaches, achieving a facial reaction correlation of 0.26 and diversity score of 0.094 while maintaining competitive realism. The code is open-sourced at \href{https://github.com/Hunan-Tiger/ReactDiff}{github}.

Figures

Figures reproduced from arXiv: 2505.14151 by the authors.

Figure 1
Figure 1. Illustration of the facial reaction generation task. Given the speaker’s behaviors (video and audio), the goal is to generate multiple listener reactions, ensuring that they are appropriate, realistic, and diverse. 1. Introduction Human-Computer Interaction (HCI) has evolved signif￾icantly since its emergence in the 1980s, expanding from simple interface designs to sophisticated emotional and be￾havioral interaction… view at source ↗
Figure 2
Figure 2. a) Overview of training process of ReactDiff. The extracted video and audio features enter the Latent Reaction Space through the Multi-Modality Transformer, in which the speaker’s behaviour constraint vector 𝐳𝑏𝑐 serves as a generation condition. b) Overview of MMT. Cross attention not only enhances the internal representation between the modalities but also aligns the semantics between the two modalities, and Transf… view at source ↗
Figure 3
Figure 3. Qualitative comparison of generated listener re￾actions by models on the testing set and 𝑛 denotes frame numbers. Speaker1 Video Reaction 1 Reaction 2 Reaction 3 n = 1 n = 150 n = 300 n = 450 n = 600 n =750 Speaker1 Audio Speaker2 Audio GT listener1 Speaker2 Video GT listener2 Reaction 1 Reaction 2 Reaction 3 n = 1 n = 150 n = 300 n = 450 n = 600 n =750 [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Two examples of generated listener reactions by ReactDiff on the test set and 𝑛 denotes frame numbers. : Preprint submitted to Elsevier Page 13 of 12 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 13 canonical work pages

  1. [13]

    Proceedings 1, Springer. pp. 707–714. Peebles,W.,Xie,S.,2023. Scalablediffusionmodelswithtransformers,in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205. Pham, L., Vu, T.H., Tran, T.A.,

    work page 2023

  2. [15]

    arXiv preprint arXiv:2208.07862

    Stylefacev: Face video generation via decomposing and recomposing pretrained stylegan3. arXiv preprint arXiv:2208.07862 . Radford, A.,

    arXiv

  3. [16]

    Introducing the recola multimodal corpus of remote collaborative and affective interactions,in:201310thIEEEinternationalconferenceandworkshops on automatic face and gesture recognition (FG), IEEE. pp. 1–8. Rombach,R.,Blattmann,A.,Lorenz,D.,Esser,P.,Ommer,B.,2022. High- resolutionimagesynthesiswithlatentdiffusionmodels,in:Proceedings of the IEEE/CVF confer...

    work page 2022

  4. [18]

    arXiv preprint arXiv:2010.02502

    Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 . Song,S.,Spitale,M.,Luo,C.,etal.,2023. React2023:thefirstmulti-modal multipleappropriatefacialreactiongenerationchallenge. arXivpreprint arXiv:2306.06583 . Song, S., Spitale, M., Luo, C., et al.,

    arXiv 2010

  5. [19]

    arXivpreprint arXiv:2401.05166

    React 2024: the second multipleappropriatefacialreactiongenerationchallenge. arXivpreprint arXiv:2401.05166 . Takeuchi, A., Nagao, K.,

    arXiv 2024

  6. [22]

    The genea challenge 2022: A large evaluation of data-driven co-speech gesture generation, in: Proceedings of the 2022 International Conference on Multimodal Interaction, pp. 736–747. Yu, J., Zhao, J., Xie, G., Chen, F., Yu, Y., Peng, L., Li, M., Dai, Z.,

    work page 2022

  7. [27]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.,2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems

    work page 2017

  8. [30]

    Denoisingdiffusionprobabilisticmodels

    Ho,J.,Jain,A.,Abbeel,P.,2020. Denoisingdiffusionprobabilisticmodels. Advances in neural information processing systems 33, 6840–6851. Hoque,X.,Mann,A.,Sharma,G.,Dhall,A.,2023. Beamer:Behavioralen- coder to generate multiple appropriate facial reactions, in: Proceedings of the 31st ACM International Conference on Multimedia, pp. 9536–

    work page 2020

Show all 22 references
  1. [193]

    Attentionisallyouneed

    Vaswani,A.,Shazeer,N.,Parmar,N.,etal.,2017. Attentionisallyouneed. Advances in neural information processing systems

    2017

  2. [1989]

    Biometrics , 255–268

    A concordance correlation coefficient to evaluate reproducibility. Biometrics , 255–268. Li,J.,Li,D.,Xiong,C.,Hoi,S.,2022. Blip:Bootstrappinglanguage-image pre-training for unified vision-language understanding and generation, in: International conference on machine learning, ...

    2022

  3. [1999]

    Seminal Graphics Papers: Pushing the Boundaries, Volume 2 URL:https://api.semanticscholar.org/CorpusID:203705211

    A morphable model for the synthesis of 3d faces. Seminal Graphics Papers: Pushing the Boundaries, Volume 2 URL:https://api.semanticscholar.org/CorpusID:203705211. Cafaro,A.,Wagner,J.,Baur,T.,etal.,2017. Thenoxidatabase:multimodal recordings of mediated novice-expert interactio...

    2017

  4. [2005]

    Intelligent expressions of emotions, in: Affective Computing and Intelligent In- teraction: First International Conference, ACII 2005, Beijing, China, October 22-24,

    2005

  5. [2013]

    arXiv preprint arXiv:1312.6114

    Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 . Kumar, P., Raman, B.,

    arXiv

  6. [2015]

    PhysicsLettersA379,680–687

    Analysis of detrended time-lagged cross-correlation betweentwononstationarytimeseries. PhysicsLettersA379,680–687. Song,H.,Zhang,W.N.,Cui,Y.,Wang,D.,Liu,T.,2019.Exploitingpersona information for diverse generation of conversational responses. arXiv preprint arXiv:1905.12188 . ...

    2019 arXiv

  7. [2016]

    arXiv preprint arXiv:1608.03983

    Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 . Lu,J.,Batra,D.,Parikh,D.,Lee,S.,2019.Vilbert:Pretrainingtask-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems

    2019 arXiv

  8. [2018]

    arXiv preprint arXiv:1810.04805

    Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . Dinh,L.,Sohl-Dickstein,J.,Bengio,S.,2016. Densityestimationusingreal nvp. arXiv preprint arXiv:1605.08803 . Dosovitskiy, A.,

    2016 arXiv

  9. [2020]

    arXiv preprint arXiv:2010.11929

    An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 . Ekman,P.,Friesen,W.V.,1978. Facialactioncodingsystem. Environmen- tal Psychology & Nonverbal Behavior . Geirhos,R.,Jacobsen,J.H.,Michaelis,C.,Zemel,R.,Brendel,W.,Bethg...

    2010 arXiv

  10. [2021]

    Facial expression recognition using residual masking network, in: Proc. Int. Conf. on Pattern Recog., IEEE. pp. 4513–4519. Poggi,I.,Pelachaud,C.,2000. Performativefacialexpressionsinanimated faces . Preechakul, K., Chatthee, N., Wizadwongsa, S., Suwajanakorn, S.,

    2000

  11. [2022]

    arXiv preprint arXiv:2205.01782

    Learning multi- dimensional edge feature-based au relation graph for facial action unit recognition. arXiv preprint arXiv:2205.01782 . Luo, C., Song, S., Xie, W., Spitale, M., Ge, Z., Shen, L., Gunes, H.,

    arXiv

  12. [2023]

    arXiv preprint arXiv:2305.15270

    Reversible graph neural network-based reaction distribution learning for multiple appropriate facial reactions generation. arXiv preprint arXiv:2305.15270 . Yoon, Y., Wolfert, P., Kucherenko, T., et al.,

    arXiv

  13. [2024]

    Vector quantized diffusion models for multiple appropriate reactions generation, in: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), IEEE. pp. 1–5. Ochs, M., Niewiadomski, R., Pelachaud, C., Sadek, D.,

    2024

  14. [2327]

    Berndt,D.J.,Clifford,J.,1994.Usingdynamictimewarpingtofindpatterns in time series, in: Proceedings of the 3rd international conference on knowledge discovery and data mining, pp. 359–370. Blanz, V., Vetter, T.,

    1994

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.