REVIEW 3 major objections 5 minor 52 references
FSGAN: Subject Agnostic Face Swapping and Reenactment
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read FSGAN establishes a subject-agnostic face swapping and reenactment pipeline: one system transfers pose, expression, and identity between any two faces without per-subject training.
desk verdict Real engineering gains in face reenactment, but a self-contradictory training description undercuts the subject-agnostic claim until clarified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The recurrent reenactment generator Gr is the load-bearing component: given a face image and a heatmap of target landmarks, it repeatedly transforms the image through interpolated intermediate landmark positions, splitting a large pose or expression change into several smaller steps instead of one risky jump. This stepwise, recurrent design is what lets a network trained only on same-subject frame pairs generalize to new identities. Around it, the system uses Delaunay triangulation and barycentric coordinates to interpolate between multiple source views, a completion generator Gc for occluded regions, and a blending generator Gb trained with a Poisson blending loss that combines the classic Poisson gradient constraint with a perceptual loss.
What would settle it
Run the system on unseen source/target pairs while sweeping the target yaw from frontal to about 90 degrees, and compare the output's face-verification similarity to the source identity; if identity similarity collapses toward the target as the pose gap grows, then the subject-agnostic claim fails precisely in the large-pose regime the paper itself identifies as a limitation.
Extended reading notes
Core claim
The paper claims that FSGAN is the first method to simultaneously manipulate pose, expression, and identity without person-specific or pair-specific training. The system consists of a recurrent reenactment generator that maps a source face onto the target pose and expression through interpolated intermediate landmarks, a segmentation network, a face inpainting network that fills occluded or missing regions, and a blending network trained with a new Poisson blending loss that combines Poisson optimization with perceptual loss. For videos, the method builds an appearance map from multiple source views using Delaunay triangulation and barycentric coordinates, allowing continuous interpolation of face views rather than relying on a single image. On held-out video pairs, it reports a pose error of 2.49 degrees and a landmark error of 22.2 pixels, lower than the compared baselines.
Load-bearing premise
The reenactment generator is trained only on pairs of frames of the same person, yet it is assumed that what it learns about moving one person's own face transfers unchanged to moving one person's face into another person's head, without any explicit separation of identity from pose and expression.
Editorial extensions
If this is right
- Face swapping and reenactment become usable on people the system has never seen, removing the expensive per-subject data collection and training required by earlier methods.
- For video, the appearance-map interpolation means an entire source video can serve as input, giving continuous pose coverage rather than requiring a single carefully chosen source frame.
- The inpainting and blending stages let the method handle occluded faces and match the transferred face to the target's skin tone and lighting, which the ablation study shows improves pose and expression retention.
- A single pipeline now covers swapping, reenactment, view interpolation, occlusion handling, and blending, making these tasks substantially more accessible to non-experts.
- The paper argues that making such synthesis accessible should accelerate the development of technical counter-measures and informed policy for detecting manipulated media.
Reading between the lines
- Because the reenactment generator is trained only on same-subject frame pairs, the system does not explicitly enforce separation of identity from pose and expression; an explicit identity-invariance constraint or adversarial identity loss could further strengthen cross-subject transfer.
- The recurrent stepwise design suggests a predictable scaling behavior: the larger the pose gap, the more intermediate steps are needed, so one could tune the number of steps automatically based on the angular difference between source and target.
- The Delaunay/barycentric view-interpolation scheme is not inherently face-specific; the same triangulation of a low-dimensional pose space could be applied to other objects, such as hands or bodies, where continuous pose control is needed.
- A direct stress test would be extreme poses and unusual expressions: if identity fidelity degrades smoothly as the target deviates from the source's pose, that would confirm the mechanism is genuinely pose-driven rather than identity-driven.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FSGAN, a face swapping and reenactment system that is claimed to be subject agnostic, meaning it can operate on pairs of faces without per-subject or per-pair training. The method comprises a recurrent reenactment generator (Gr) that iteratively adjusts a source face to a target pose and expression using landmark heatmaps, a segmentation network (Gs), a face inpainting network (Gc), and a face blending network (Gb) trained with a novel Poisson blending loss. Training uses IJB-C video data, and evaluation is done on FaceForensics++ against DeepFakes, Nirkin et al., and Face2Face. The paper reports quantitative results for identity verification, SSIM, pose error, and landmark error, plus an ablation study and qualitative examples.
Significance. If validated, the subject-agnostic claim is practically significant because it removes the need for expensive per-subject data collection and training, making face swapping and reenactment accessible for a wide range of applications. The modular pipeline, the recurrent stepwise reenactment mechanism for large pose changes, and the proposed Poisson blending loss are useful technical contributions. The paper provides quantitative comparisons and an ablation study on an external benchmark, which is a strength, and it also acknowledges limitations such as degradation under large yaw angles and resolution constraints. However, the central subject-agnostic claim depends on training details that are currently ambiguous, so the contribution is defensible but not yet fully established.
major comments (3)
- [Sec. 3.2, Eq. (7); Sec. 4.1] placeholder
- [Sec. 3.2, Eq. (6)-(7)] placeholder
- [Table 1, Sec. 5.4] placeholder
minor comments (5)
- [Sec. 3.2] placeholder
- [Eq. (11)] placeholder
- [Supplementary Material, Fig. 9 caption] placeholder
- [Sec. 3.3] placeholder
- [Table 2] placeholder
Circularity Check
No significant circularity: FSGAN's subject-agnostic claim is tested on external FaceForensics++ benchmarks, and self-citations are non-load-bearing baselines or component-level resources.
full rationale
FSGAN's central claim of subject-agnostic reenactment and swapping is supported by an end-to-end trainable pipeline and is evaluated on FaceForensics++ videos not seen during training (Sec. 5.2, Fig. 6). The reported quantitative metrics (Table 1) measure pose and expression retention on that external benchmark; lower landmark error follows from conditioning the reenactment generator on target landmark heatmaps, which is a standard functional objective, not a fitted parameter renamed as a prediction. The self-citations to Nirkin et al. [35] appear as a baseline for comparison and as a source of segmentation training data; neither is load-bearing for the subject-agnostic claim. The inconsistency between Sec. 3.2's same-subject training pairs and Sec. 4.1's 'two randomly chosen subjects' is a reproducibility and generalization ambiguity, not a circular step: the model is not defined in terms of its evaluation metric, and no equation reduces to its own input by construction. The limitations stated in Sec. 6 (angular degradation, resolution limits, sparse landmark tracking) are acknowledged weaknesses, not circular reasoning. I find no step where a claimed prediction is equivalent to an input, a fitted parameter, or a self-citation chain.
Assumptions & free parameters
free parameters (6)
- perceptual loss weight lambda_perc =
1
- pixel loss weight lambda_pixel =
0.1
- adversarial loss weight lambda_adv =
0.001
- segmentation loss weight lambda_seg =
0.1
- reenactment steps n
- Adam learning rate =
0.0002
assumptions (4)
- domain assumption dlib facial landmarks and Euler angles capture pose and expression sufficiently for all subjects
- domain assumption Same-subject training pairs generalize to cross-subject reenactment
- domain assumption IJB-C training videos provide sufficient pose and expression coverage
- domain assumption VGG perceptual networks pretrained on VGGFace2 and CelebA are suitable face-similarity measures
Cite this review
Pith. "Pith review of FSGAN: Subject Agnostic Face Swapping and Reenactment." pith.science (2026). https://pith.science/paper/UJ5EV6NW
@misc{pith2026190805932,
author = {Pith},
title = {Pith review of: FSGAN: Subject Agnostic Face Swapping and Reenactment},
year = {2026},
howpublished = {\url{https://pith.science/paper/UJ5EV6NW}},
note = {Machine review of arXiv:1908.05932}
}
read the original abstract
We present Face Swapping GAN (FSGAN) for face swapping and reenactment. Unlike previous work, FSGAN is subject agnostic and can be applied to pairs of faces without requiring training on those faces. To this end, we describe a number of technical contributions. We derive a novel recurrent neural network (RNN)-based approach for face reenactment which adjusts for both pose and expression variations and can be applied to a single image or a video sequence. For video sequences, we introduce continuous interpolation of the face views based on reenactment, Delaunay Triangulation, and barycentric coordinates. Occluded face regions are handled by a face completion network. Finally, we use a face blending network for seamless blending of the two faces while preserving target skin color and lighting conditions. This network uses a novel Poisson blending loss which combines Poisson optimization with perceptual loss. We compare our approach to existing state-of-the-art systems and show our results to be both qualitatively and quantitatively superior.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Creating a photoreal digital actor: The digital emily project
Oleg Alexander, Mike Rogers, William Lambeth, Matt Chi- ang, and Paul Debevec. Creating a photoreal digital actor: The digital emily project. In Conf. Visual Media Production, pages 176–187. IEEE, 2009
work page 2009
-
[2]
Martin Arjovsky, Soumith Chintala, and L ´eon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017
arXiv 2017
-
[3]
Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen. Bringing portraits to life. ACM Transac- tions on Graphics (TOG), 36(6):196, 2017
work page 2017
-
[4]
Face swapping: automatically replacing faces in photographs
Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter Bel- humeur, and Shree K Nayar. Face swapping: automatically replacing faces in photographs. ACM Trans. on Graphics , 27(3):39, 2008
work page 2008
-
[5]
Face identification across different poses and illuminations with a 3d morphable model
V olker Blanz, Sami Romdhani, and Thomas Vetter. Face identification across different poses and illuminations with a 3d morphable model. In Int. Conf. on Automatic Face and Gesture Recognition, pages 192–197, 2002
work page 2002
-
[6]
V olker Blanz, Kristina Scherbaum, Thomas Vetter, and Hans-Peter Seidel. Exchanging faces in images. Comput. Graphics F orum, 23(3):669–676, 2004
work page 2004
-
[7]
Face recognition based on fitting a 3d morphable model
V olker Blanz and Thomas Vetter. Face recognition based on fitting a 3d morphable model. Trans. Pattern Anal. Mach. Intell., 25(9):1063–1074, 2003
work page 2003
-
[8]
Robust face landmark estimation under occlusion
Xavier P Burgos-Artizzu, Pietro Perona, and Piotr Doll ´ar. Robust face landmark estimation under occlusion. In Proc. Int. Conf. Comput. Vision, pages 1513–1520. IEEE, 2013
work page 2013
Show all 52 references
-
[9]
Vggface2: A dataset for recognising faces across pose and age
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 67–74. IEEE, 2018
2018
-
[10]
FacePoseNet: Making a case for landmark-free face alignment
Feng-Ju Chang, Anh Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, and G´erard Medioni. FacePoseNet: Making a case for landmark-free face alignment. In Proc. Int. Conf. Com- put. Vision Workshops, 2017
2017
-
[11]
Deep, landmark-free fame: Face alignment, modeling, and expression estimation
Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, and G ´erard Medioni. Deep, landmark-free fame: Face alignment, modeling, and expression estimation. Int. J. Comput. Vision, 127(6-7):930–956, 2019
2019
-
[12]
FaceSwap
DeepFakes. FaceSwap. https://github.com/ deepfakes/faceswap. Accessed: 2019-02-06
2019
-
[13]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014
2014
-
[14]
Effective face frontalization in unconstrained images
Tal Hassner, Shai Harel, Eran Paz, and Roee Enbar. Effective face frontalization in unconstrained images. In Proc. Conf. Comput. Vision Pattern Recognition, 2015
2015
-
[15]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[16]
Computer graphics: principles and prac- tice
John F Hughes, Andries Van Dam, James D Foley, and Steven K Feiner. Computer graphics: principles and prac- tice. Pearson Education, 2014
2014
-
[17]
Image-to-image translation with conditional adver- sarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1125–1134, 2017
2017
-
[18]
Perceptual losses for real-time style transfer and super-resolution
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision , pages 694–711. Springer, 2016
2016
-
[19]
Augmenting crfs with boltzmann machine shape pri- ors for image labeling
Andrew Kae, Kihyuk Sohn, Honglak Lee, and Erik Learned- Miller. Augmenting crfs with boltzmann machine shape pri- ors for image labeling. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2019–2026, 2013
2019
-
[20]
Progressive growing of gans for improved quality, stability, and variation
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017
2017 arXiv
-
[21]
Transfiguring portraits
Ira Kemelmacher-Shlizerman. Transfiguring portraits. ACM Trans. on Graphics, 35(4):94, 2016
2016
-
[22]
Figure 8: Ablation study
Hyeongwoo Kim, Pablo Carrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick P ´erez, Chris- tian Richardt, Michael Zollh ¨ofer, and Christian Theobalt. Figure 8: Ablation study. From columns 3 and 5, without the completion network,Gc, the transferred face ...
2018
-
[23]
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755–1758, 2009
2009
-
[24]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
-
[25]
Photo- realistic single image super-resolution using a generative ad- versarial network
Christian Ledig, Lucas Theis, Ferenc Husz´ar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo- realistic single image super-resolution using a generative ad- versarial network. In Proceedings of the IE...
2017
-
[26]
Face swapping under large pose variations: A 3D model based approach
Yuan Lin, Shengjin Wang, Qian Lin, and Feng Tang. Face swapping under large pose variations: A 3D model based approach. In Int. Conf. on Multimedia and Expo , pages 333–
-
[27]
Large-scale celebfaces attributes (celeba) dataset
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018
2018
-
[28]
Least squares generative adversarial networks
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE Interna- tional Conference on Computer Vision , pages 2794–2802, 2017
2017
-
[29]
Face-specific data augmentation for un- constrained face recognition
Iacopo Masi, Anh Tun Trn, Tal Hassner, Gozde Sahin, and G´erard Medioni. Face-specific data augmentation for un- constrained face recognition. Int. J. Comput. Vision , 127(6- 7):642–667, 2019
2019
-
[30]
Iarpa janus benchmark-c: Face dataset and protocol
Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. Iarpa janus benchmark-c: Face dataset and protocol. In 2018 Inter- national Conference on Biometrics (ICB) , pages 158–165. I...
2018
-
[31]
Conditional generative adversarial nets
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014
2014 arXiv
-
[32]
Pho- torealistic face de-identification by aggregating donors face components
Saleh Mosaddegh, Loic Simon, and Fr ´ed´eric Jurie. Pho- torealistic face de-identification by aggregating donors face components. In Asian Conf. Comput. Vision, pages 159–174. Springer, 2014
2014
-
[33]
Fsnet: An identity-aware generative model for image-based face swapping
Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. Fsnet: An identity-aware generative model for image-based face swapping. In Proc. of Asian Conference on Computer Vision (ACCV). Springer, Dec 2018
2018
-
[34]
Rsgan: face swapping and editing using face and hair repre- sentation in latent spaces
Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. Rsgan: face swapping and editing using face and hair repre- sentation in latent spaces. arXiv preprint arXiv:1804.03447, 2018
2018 arXiv
-
[35]
On face segmentation, face swapping, and face perception
Yuval Nirkin, Iacopo Masi, Anh Tran Tuan, Tal Hassner, and Gerard Medioni. On face segmentation, face swapping, and face perception. In Automatic Face & Gesture Recognition (FG 2018), 2018 13th IEEE International Conference on , pages 98–105. IEEE, 2018
2018
-
[36]
Pois- son image editing
Patrick P ´erez, Michel Gangnet, and Andrew Blake. Pois- son image editing. ACM Transactions on graphics (TOG) , 22(3):313–318, 2003
2003
-
[37]
Ganimation: Anatomically-aware facial animation from a single image
Albert Pumarola, Antonio Agudo, Aleix M Martinez, Al- berto Sanfeliu, and Francesc Moreno-Noguer. Ganimation: Anatomically-aware facial animation from a single image. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 818–833, 2018
2018
-
[38]
U- net: Convolutional networks for biomedical image segmen- tation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In International Conference on Medical image com- puting and computer-assisted intervention , pages 234–241. Springer, 2015
2015
-
[39]
Faceforen- sics++: Learning to detect manipulated facial images
Andreas R ¨ossler, Davide Cozzolino, Luisa Verdoliva, Chris- tian Riess, Justus Thies, and Matthias Nießner. Faceforen- sics++: Learning to detect manipulated facial images. arXiv, 2019
2019
-
[40]
Triple consistency loss for pairing distributions in gan-based face synthesis
Enrique Sanchez and Michel Valstar. Triple consistency loss for pairing distributions in gan-based face synthesis. arXiv preprint arXiv:1811.03492, 2018
2018 arXiv
-
[41]
Very deep convo- lutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
2014 arXiv
-
[42]
Synthesizing obama: learn- ing lip sync from audio
Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learn- ing lip sync from audio. ACM Transactions on Graphics (TOG), 36(4):95, 2017
2017
-
[43]
Figaro, hair detection and seg- mentation in the wild
Michele Svanera, Umar Riaz Muhammad, Riccardo Leonardi, and Sergio Benini. Figaro, hair detection and seg- mentation in the wild. In 2016 IEEE International Con- ference on Image Processing (ICIP) , pages 933–937. IEEE, 2016
2016
-
[44]
Face2face: Real-time face capture and reenactment of rgb videos
Justus Thies, Michael Zollhofer, Marc Stamminger, Chris- tian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2387–2395, 2016
2016
-
[45]
Cr-gan: learning complete representations for multi-view generation
Yu Tian, Xi Peng, Long Zhao, Shaoting Zhang, and Dim- itris N Metaxas. Cr-gan: learning complete representations for multi-view generation. arXiv preprint arXiv:1806.11191, 2018
2018 arXiv
-
[46]
Extreme 3D face reconstruc- tion: Looking past occlusions
Anh Tuan Tran, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and G ´erard Medioni. Extreme 3D face reconstruc- tion: Looking past occlusions. In Proc. Conf. Comput. Vision Pattern Recognition, 2018
2018
-
[47]
High-resolution image syn- thesis and semantic manipulation with conditional gans
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image syn- thesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
2018
-
[48]
An eye for an eye: A single camera gaze-replacement method
Lior Wolf, Ziv Freund, and Shai Avidan. An eye for an eye: A single camera gaze-replacement method. In Proc. Conf. Comput. Vision Pattern Recognition , pages 817–824. IEEE, 2010
2010
-
[49]
Gp-gan: Towards realistic high-resolution image blending
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. Gp-gan: Towards realistic high-resolution image blending. arXiv preprint arXiv:1703.07195, 2017
2017 arXiv
-
[50]
Reenactgan: Learning to reenact faces via boundary transfer
Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy. Reenactgan: Learning to reenact faces via boundary transfer. In ECCV, Sept. 2018
2018
-
[51]
Seman- tic image inpainting with deep generative models
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Seman- tic image inpainting with deep generative models. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5485–5493, 2017
2017
-
[52]
Unpaired image-to-image translation using cycle- consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 2223– 2232, 2017. Figure 9: Additional qualitative fac...
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.