REVIEW 4 major objections 5 minor 43 references
A Realistic Face-to-Face Conversation System based on Deep Neural Networks
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Two sequence-to-sequence models and a GAN synthesizer can predict a listener's facial reactions and render them as realistic avatar images from the speaker's facial action units and head pose alone.
desk verdict Useful listening-phase model that cannot hear, wrapped in an evaluation that never measures naturalness; worth reviewing, but the central claim needs real support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key representation is the AU+POSE vector, a 20-dimensional code of 17 facial action units and 3 head pose angles extracted from each face frame; it is the common currency of the whole pipeline, used as input and output of the listening and speaking sequence-to-sequence models and as the conditioning signal for the synthesizer. Two loss terms carry the argument. The continuity loss penalizes the largest mismatch between predicted and ground-truth adjacent-frame differences over a window, which is what suppresses the jerkiness that pure MSE leaves in generated sequences. The synthesizer's temporal-smoothing adversarial loss lets the generator condition on the previously synthesized frame, so consecutive outputs stay coherent; the AU+POSE values are painted into the center of a blank image so the GAN receives spatial coordinate constraints.
What would settle it
Take two video clips in which the speaker's AU+POSE trajectories are nearly identical but the spoken words differ sharply, such as a compliment versus a question; if the listening model produces nearly identical predicted listener reactions in both, the claim that it generates natural conversational reactions is refuted.
Extended reading notes
Core claim
The central claim is that natural face-to-face conversation can be driven entirely by a compact facial feature representation, without explicit 3D models. The listening network takes ten frames of the speaker's 20-dimensional AU+POSE vector (17 facial action units plus 3 head pose angles) and outputs the listener's AU+POSE sequence; the speaking network takes word embeddings of the response sentence and outputs the new speaker's AU+POSE sequence. Both are LSTM sequence-to-sequence models trained with MSE plus a continuity loss that compares adjacent-frame differences between prediction and ground truth. The synthesizer, based on a Pix2PixHD-style GAN with a temporal smoothing adversarial loss, maps these AU+POSE streams to face images. Evaluated on talk-show data, the continuity-trained models outperform the MSE-only baseline on MSE and cosine similarity, and the synthesizer's images reconstruct action units and pose with average errors below 0.105 for action units and 0.073 for pose. The paper concludes that the system generates natural facial reactions and realistic facial images.
Load-bearing premise
The listening model assumes a listener's nonverbal reaction is fully determined by the speaker's facial action units and head pose over the previous ten frames, leaving no role for what is actually said, how it is said, or anything that happened earlier in the conversation.
Editorial extensions
If this is right
- An avatar built this way can react while the user is talking, because the listening model converts the speaker's AU+POSE stream into listener reactions frame by frame.
- A text-only response suffices to drive the avatar's speaking face, since the speaking model maps word embeddings to AU+POSE and the synthesizer renders them.
- The continuity loss is what makes the predicted sequences smooth enough to look natural, and its benefit is larger in the speaking phase, where action-unit and pose variance is higher.
- The synthesized images preserve the conditioning action units and head pose well enough that re-extracted features stay close to the original (average errors 0.046 and 0.038 for listening, 0.105 and 0.073 for speaking).
- The whole pipeline needs only two-person video and text, not 3D scans or motion capture, so it can be retrained on new domains with similar data.
Reading between the lines
- If the AU+POSE representation is sufficiently expressive, the same listening-and-speaking loop could be retrained on other dyadic settings—interviews, telehealth, tutoring—where two-person video exists; the paper only demonstrates one data domain.
- The continuity loss, which penalizes the maximum adjacent-frame difference error over a window, is a generic temporal smoothness prior and could transfer to gesture or body-motion synthesis, though the paper does not investigate that.
- A decisive test of the listening model's ceiling is to feed identical speaker AU+POSE sequences paired with very different speech content; if the predicted listener reactions are nearly identical, the model is content-blind by construction.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a face-to-face conversation system composed of two sequence-to-sequence models (one for listening, one for speaking) and a Pix2PixHD-based face synthesizer. The listening model maps a 10-frame window of the speaker's AU+POSE features to the listener's AU+POSE features; the speaking model maps response text to the speaker's AU+POSE; and the synthesizer renders AU+POSE into face images. The seq2seq models are trained with an MSE loss augmented by a continuity loss. Experiments on ESPN show data compare the proposed training to an MSE-only baseline using MSE and cosine similarity, evaluate the synthesizer by re-extracting AU+POSE from generated images, and provide qualitative comparisons with a 3D avatar. The authors claim that the system generates natural facial reactions and realistic facial images.
Significance. If the claims were substantiated, the system would be a useful step toward realistic avatar-based conversation without 3D scanning or motion capture. The continuity loss and the complete two-phase pipeline are plausible contributions, and the use of a large real-world conversational dataset is positive. However, the current evidence is insufficient: the listening model ignores speech content, the quantitative metrics do not measure naturalness and lack statistical substantiation, the synthesizer evaluation is circular, and the end-to-end evaluation is qualitative. These issues affect the central claim and would need to be addressed before the paper could be accepted.
major comments (4)
- [Section 3.3, Eq. (1)] The listening model is architecturally mismatched with the paper's own description of conversation. Its input is only the speaker's 17-D action units and 3-D head pose over 10 frames, yet Section 1 states that the listener receives verbal information as well as nonverbal cues and gives nonverbal feedback. Nodding at a question, laughing at a joke, or reacting to surprising information depends on lexical and prosodic content. Therefore the model cannot generate content-appropriate listener reactions, and the abstract's central claim that the system produces natural facial reactions is not supported by the design and is not tested by the experiments in Section 4.
- [Section 4.3, Eq. (10) and Table 3] The synthesizer evaluation is circular. The generated images are conditioned on AU+POSE values, and the evaluation re-extracts AU+POSE from the generated images and compares them to the conditioning values. This measures reconstruction fidelity, not the realism or naturalness of the images, and it does not validate the quality of the predicted reaction sequences. No comparison against the 3D avatar baseline is performed using this metric, so the claim that the synthesizer produces realistic images is not established.
- [Section 4.2, Tables 1 and 2] The reported MSE and cosine values are point estimates without variance, confidence intervals, or test-set sizes. The listening-phase improvement is small (MSE 0.0565 vs 0.0540; cosine 0.983 vs 0.992), so without error bars or significance testing it is impossible to determine whether the proposed continuity loss provides a real advantage over the MSE-only baseline. The tables should include standard deviations or confidence intervals and the number of test samples.
- [Section 4.4 and Figures 5-6] The end-to-end evaluation is qualitative, with only four examples and no user study, perceptual metric, or quantitative comparison. The statement that the synthesized images are 'more realistic' than the 3D avatar is based on subjective visual inspection and does not substantiate the abstract's claims of natural facial reactions and realistic facial images. A user study or a well-defined perceptual metric would be needed to support these claims.
minor comments (5)
- [Section 4.3, Eq. (10)] There is a typo in the equation: 'reocnstructed' should be 'reconstructed'.
- [Section 4.2, Eq. (9)] The cosine similarity is defined as dcon in Eq. (9), but it is referred to as 'Cosine' in the table headers; the notation is inconsistent.
- [Section 3.3, Eq. (2)] There is a typo: 'AU+PUSE' should be 'AU+POSE'.
- [Section 2] The relationship to the prior work by Chu et al. [5] is described only briefly; the authors should clarify what is new in their listening model and why the continuity loss is necessary in addition to MSE.
- [Section 4.1] The dataset description is minimal. The number of videos is given, but the train/test split, number of frames, and subject details are not reported, which hinders reproducibility.
Circularity Check
The synthesizer realism evaluation in §4.3 is a closed loop: it re-extracts the same AU+POSE features that were fed in as conditioning, so Table 3 verifies input preservation rather than naturalness.
-
self definitional
[Section 3.6 ('Face synthesizer') and Section 4.3 ('Evaluation on avatar synthesizer', Eq. 10, Table 3).]
"With the AU+POSE as input, the face synthesizer aims to generate face images of the target person with corresponding postures and expressions. ... We run facial action units and head pose extractor on the output of our avatar synthesizer, and compare these reconstructed action units and poses to the counterpart of the original input video."
The synthesizer is conditioned on an AU+POSE image x, and its training losses (L1, VGG, GAN) align G(x) with ground-truth y for that x. The evaluation metric Eq. (10) then re-extracts AU+POSE from G(x) and compares it to the same AU+POSE values that were used to build x. Low d_au/d_pose therefore only show that the renderer preserves its conditioning input, not that the resulting face image is realistic or that the conversation is natural. The conclusion that the system 'can generate natural and realistic avatars' thus relies in part on a closed-loop consistency check rather than an independent measure of realism. The seq2seq listening/speaking evaluations are separate and not circular.
full rationale
This paper is an empirical system paper rather than a derivation, so most of its chain is ordinary supervised learning. The listening and speaking model evaluations compare predicted AU+POSE sequences against held-out listener/speaker ground truth while conditioning on the other party's input; that is an independent test of the seq2seq component and is not circular. The one genuinely circular component is the avatar-synthesizer evaluation in §4.3: because the synthesizer's input is exactly the AU+POSE representation that is later re-extracted from the output and used as the error criterion, the reported AU/Pose errors verify input reconstruction rather than realism or naturalness. The additional weakness that the listening model assumes listener reactions are a deterministic function of a 10-frame speaker AU+POSE window is a modeling-limitation/correctness concern, not a circularity, since the model is still trained and evaluated on independent ground-truth listener data. Self-citations ([21], [22], [39]) appear only as motivational references and are not load-bearing, and no uniqueness theorem or imported ansatz is used to force the architecture. Therefore the central seq2seq claims retain independent content, while the realism claim is partially supported by a self-referential metric, yielding a moderate rather than severe circularity score.
Assumptions & free parameters
free parameters (2)
- alpha (continuity loss weight) =
0.1
- gamma (head-pose weight in MSE loss) =
8.1
assumptions (5)
- domain assumption OpenFace AU+POSE estimates are accurate enough to serve as ground truth for natural facial reactions.
- domain assumption A speaker's AU+POSE over a 10-frame window is sufficient to determine the listener's AU+POSE response.
- domain assumption ESPN debate show footage is representative of natural face-to-face conversation.
- domain assumption A separate conversation module generates the response sentence and is assumed to work.
- standard math LSTM and GAN training converge to models whose losses correspond to behavioral fidelity.
Cite this review
Pith. "Pith review of A Realistic Face-to-Face Conversation System based on Deep Neural Networks." pith.science (2026). https://pith.science/paper/KTW3T4CQ
@misc{pith2026190807750,
author = {Pith},
title = {Pith review of: A Realistic Face-to-Face Conversation System based on Deep Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/KTW3T4CQ}},
note = {Machine review of arXiv:1908.07750}
}
read the original abstract
To improve the experiences of face-to-face conversation with avatar, this paper presents a novel conversation system. It is composed of two sequence-to-sequence models respectively for listening and speaking and a Generative Adversarial Network (GAN) based realistic avatar synthesizer. The models exploit the facial action and head pose to learn natural human reactions. Based on the models' output, the synthesizer uses the Pixel2Pixel model to generate realistic facial images. To show the improvement of our system, we use a 3D model based avatar driving scheme as a reference. We train and evaluate our neural networks with the data from ESPN shows. Experimental results show that our conversation system can generate natural facial reactions and realistic facial images.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[35]
O. Vinyals and Q. V . Le. A neural conversational model. CoRR, abs/1506.05869, 2015. 2
arXiv 2015
-
[5]
H. Chu, D. Li, and S. Fidler. A face-to-face neural conversa- tion model. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7113–7121, 2018. 1, 2
work page 2018
-
[1]
T. Baltrusaitis, P. Robinson, and L. Morency. Openface: an open source facial behavior analysis tool. In Proc. Workshop on Applications of Computer Vision (WACV), IEEE , 2016. 3
work page 2016
-
[2]
C. Chan, S. Ginosar, T. Zhou, and A. A. Efros. Everybody dance now. arXiv preprint arXiv:1808.07371, 2018. 2, 4
arXiv 2018
-
[3]
C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):27, 2011. 3
work page 2011
-
[4]
H.-Y . Chen et al. Tensorflow–a system for large-scale ma- chine learning. In OSDI, volume 16, pages 265–283, 2016. 5
work page 2016
-
[6]
N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In international Conference on computer vision & Pattern Recognition (CVPR’05) , volume 1, pages 886–893. IEEE Computer Society, 2005. 3
work page 2005
-
[7]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 4
arXiv 2018
Show all 43 references
-
[8]
Devries, K
T. Devries, K. Biswaranjan, and G. W. Taylor. Multi-task learning of facial landmarks and expression. In Computer & Robot Vision, 2014. 2
2014
-
[9]
Y . Fan, Y . Qian, F.-L. Xie, and F. K. Soong. Tts synthe- sis with bidirectional lstm based recurrent neural networks. In Fifteenth Annual Conference of the International Speech Communication Association, 2014. 2
2014
-
[10]
Goodfellow, J
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014. 2
2014
-
[11]
Graves, A.-r
A. Graves, A.-r. Mohamed, and G. Hinton. Speech recogni- tion with deep recurrent neural networks. In 2013 IEEE in- ternational conference on acoustics, speech and signal pro- cessing, pages 6645–6649. IEEE, 2013. 2
2013
-
[12]
Hinton, L
G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen, B. Kingsbury, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012. 2
2012
-
[13]
Hochreiter and J
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 2, 4
1997
-
[14]
L. Hu, S. Saito, L. Wei, K. Nagano, J. Seo, J. Fursund, I. Sadeghi, C. Sun, Y .-C. Chen, and H. Li. Avatar digiti- zation from a single image for real-time rendering. ACM Transactions on Graphics (TOG), 36(6):195, 2017. 1
2017
-
[15]
Isola, J.-Y
P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017. 2
2017
-
[16]
Johansson, G
M. Johansson, G. Skantze, and J. Gustafson. Head pose pat- terns in multiparty human-robot team-building interactions. In International conference on social robotics , pages 351–
-
[17]
Katzenmaier, R
M. Katzenmaier, R. Stiefelhagen, and T. Schultz. Identify- ing the addressee in human-human-robot interactions based on head pose and speech. In Proceedings of the 6th interna- tional conference on Multimodal interfaces , pages 144–151. ACM, 2004. 2, 3
2004
-
[18]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. 5, 6
2014 arXiv
-
[19]
J. J. Lien, T. Kanade, J. F. Cohn, and C.-C. Li. Automated facial expression recognition based on facs action units. In Proceedings Third IEEE International Conference on Auto- matic Face and Gesture Recognition , pages 390–395. IEEE,
-
[20]
J. J.-J. Lien, T. Kanade, J. F. Cohn, and C.-C. Li. Detection, tracking, and classification of action units in facial expres- sion. Robotics and Autonomous Systems , 31(3):131–146,
-
[21]
Z. Liu, Z. Chen, J. Bai, S. Li, and S. Lian. Facial pose estima- tion by deep learning from label distributions.arXiv preprint arXiv:1904.13102, 2019. 2
1904 arXiv
-
[22]
Z. Liu, H. Hu, Z. Wang, K. Wang, J. Bai, and S. Lian. Video synthesis of human upper body with realistic face. arXiv preprint arXiv:1908.06607, 2019. 2
1908 arXiv
-
[23]
Matsuyama, A
Y . Matsuyama, A. Bhardwaj, Z. Ran, O. Romeo, S. Akoju, and J. Cassell. Socially-aware animated intelligent personal assistant agent. In Meeting of the Special Interest Group on Discourse & Dialogue, 2016. 1
2016
-
[24]
Mikolov, K
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient es- timation of word representations in vector space. Computer Science, 2013. 4
2013
-
[25]
Nickel and R
K. Nickel and R. Stiefelhagen. Visual recognition of point- ing gestures for human–robot interaction. Image and vision computing, 25(12):1875–1884, 2007. 2
2007
-
[26]
Pennington, R
J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language pro- cessing (EMNLP), pages 1532–1543, 2014. 4
2014
-
[27]
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word rep- resentations. CoRR, abs/1802.05365, 2018. 4
2018 arXiv
-
[28]
Sangineto
E. Sangineto. Pose and expression independent facial land- mark localization using dense-surf and the hausdorff dis- tance. IEEE transactions on pattern analysis and machine intelligence, 35(3):624–638, 2013. 3
2013
-
[29]
Seemann, K
E. Seemann, K. Nickel, and R. Stiefelhagen. Head pose es- timation using stereo vision for human-robot interaction. In Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings. , pages 626–631. IEEE, 2004. 2
2004
-
[30]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 4
2014 arXiv
-
[31]
Stiefelhagen, C
R. Stiefelhagen, C. Fugen, R. Gieselmann, H. Holzapfel, K. Nickel, and A. Waibel. Natural human-robot interaction using speech, head pose and gestures. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), volume 3, pages 2422–
2004
-
[32]
Subramanian, Y
R. Subramanian, Y . Yan, J. Staiano, O. Lanz, and N. Sebe. On the relationship between head pose, social attention and personality prediction for unstructured and dynamic group interactions. In Proceedings of the 15th ACM on Interna- tional conference on multimodal interaction...
-
[33]
Sutskever, O
I. Sutskever, O. Vinyals, and Q. V . Le. Sequence to sequence learning with neural networks. In Advances in neural infor- mation processing systems, pages 3104–3112, 2014. 2, 3, 4, 6
2014
-
[34]
Y .-I. Tian, T. Kanade, and J. F. Cohn. Recognizing ac- tion units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence , 23(2):97–115,
-
[36]
Waldherr, R
S. Waldherr, R. Romero, and S. Thrun. A gesture based interface for human-robot interaction. Autonomous Robots, 9(2):151–173, 2000. 2
2000
-
[37]
Wang, M.-Y
T.-C. Wang, M.-Y . Liu, J.-Y . Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro. Video-to-video synthesis. arXiv preprint arXiv:1808.06601, 2018. 2
2018 arXiv
-
[38]
Wang, M.-Y
T.-C. Wang, M.-Y . Liu, J.-Y . Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 8798–8807, 2018. 2, 4
2018
-
[39]
Z. Wang, Z. Liu, Z. Chen, H. Hu, and S. Lian. A neural virtual anchor synthesizer based on seq2seq and gan models. arXiv preprint arXiv:1908.07262, 2019. 2
1908 arXiv
-
[40]
Zen and H
H. Zen and H. Sak. Unidirectional long short-term mem- ory recurrent neural network with recurrent output layer for low-latency speech synthesis. In 2015 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 4470–4474. IEEE, 2015. 2
2015
-
[41]
L. Zhang. Automatic adaptation of a face model using action units for semantic coding of videophone sequences. IEEE Transactions on Circuits and Systems for Video Technology , 8(6):781–795, 1998. 2
1998
-
[42]
Zhang, M
Y . Zhang, M. Brady, and S. Smith. Segmentation of brain mr images through a hidden markov random field model and the expectation-maximization algorithm. IEEE transactions on medical imaging, 20(1):45–57, 2001. 2
2001
-
[43]
J.-Y . Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image- to-image translation using cycle-consistent adversarial net- works. In Proceedings of the IEEE International Conference on Computer Vision, pages 2223–2232, 2017. 2
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.