Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Video synthesis of human upper body with realistic face

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims a three-stage GAN pipeline that transfers a source video's upper-body motion, facial expression, and pose onto a target person, using facial action units and poses as identity-free intermediate representations.

desk verdict A plausible three-stage pipeline for upper-body video transfer, but the evidence is a single paragraph with no metrics, baselines, or ablations; the central claim is not established. read the letter →

arxiv 1908.06607 v3 pith:HPA3OJ47 submitted 2019-08-19 cs.CV

Zhaoxiang Liu , Huan Hu , Zipeng Wang , Kai Wang , Jinqiang Bai , Shiguo Lian This is my paper · ORCID
classification cs.CV
keywords videosynthesisgenerativeadversarialnetworkupperbodymotiontransferfacialactionunitslandmarkgenerationtemporalcoherenceposefacereenactment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a single pipeline can transfer an entire upper-body performance—body motion, pose, and facial expression—from one person in a video to another person, with the target person's own appearance and a face that stays realistic. The proposed route avoids directly translating source pixels into target pixels: it first reduces the source video to upper-body keypoints plus facial action units and head pose, converts those action units into the target person's facial landmarks, then synthesizes the target video from the keypoints and landmarks. If the claim holds, realistic reenactment no longer needs 3D motion capture or a separate face-transfer network stitched onto a body-transfer network, and identity-independent facial action units become a workable bridge between two people's faces.

What carries the argument

The central machinery is the FAUP representation—facial action units plus head pose—used as an identity-free intermediate between source and target. A first pix2pixHD-based generator maps a source FAUP vector, placed in the center of an otherwise empty image, to the target person's facial landmarks; these are concatenated with the source's upper-body keypoints, linearly normalized to the target's body shape, and fed to a second pix2pixHD generator. Temporal coherence is obtained by conditioning each generated frame on the previous generated frames and optimizing a temporal smoothing adversarial loss together with feature-matching and VGG perceptual losses.

What would settle it

Run the full pipeline on a source video whose facial action unit values lie far outside the target person's training distribution, then measure whether the synthesized target facial landmarks reproduce the source expression (for example, by comparing landmark displacement under the same action units); if the landmarks wash out or drift, the claimed expression consistency fails.

Watch

Extended reading notes

Core claim

The central claim is that the three-stage pipeline—source keypoint and action-unit extraction, target keypoint normalization with action-unit-to-landmark synthesis, and temporally smoothed image generation—produces a target-person video whose body motion, facial expression, and pose match the source sequence while the face remains photorealistic. The key move is replacing facial landmarks as the transfer medium with facial action units and pose (FAUP), which carry expression information without identity-specific spatial coordinates. This lets the same source FAUP values be mapped once into the target person's landmark space and then rendered by a GAN, rather than trying to normalize the source person's landmarks into the target's geometry directly.

Load-bearing premise

The facial-action-unit-to-landmark generator is trained only on the target person's own facial action unit and landmark pairs, so it must successfully transfer to the source person's action unit values even when those values are outside the target's training range.

Editorial extensions

If this is right

  • Body motion and facial expression are transferred in the same pipeline, so there is no seam where separately generated face and body images meet.
  • Because FAUP values carry no spatial coordinates and are not tied to identity, the source's expression can be used directly without facial landmark normalization.
  • Sequential frame conditioning with the temporal smoothing loss is expected to reduce flicker compared with per-frame image translation.
  • The method needs only trained 2D pose and action-unit extractors plus paired target data, not 3D motion capture or a multi-person face model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An inference the paper does not draw: the FAUP-to-landmark mapping is only as reliable as the target person's training coverage, so source expressions outside that range may break the consistency claim; augmenting the target data with varied FAUP values would be the natural test.
  • Extending the same logic to full-body transfer is straightforward because the identity-free intermediate is not specific to upper-body keypoints; only the keypoint normalization would change.
  • A quantitative check the paper leaves for later: measuring landmark alignment or perceptual similarity between synthesized frames and target ground truth under matched expressions would place a bound on how far the photorealistic claim extends beyond the two example sequences.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a three-stage GAN-based pipeline for transferring upper-body motion, pose, and facial expression from a source video to a target person. Stage 1 extracts upper body keypoints (UBKP) and facial action units plus pose (FAUP) from the source video. Stage 2 normalizes the UBKP between source and target, and uses a pix2pixHD-style generator to map source FAUP into target facial landmarks. Stage 3 concatenates the normalized UBKP and generated landmarks into an image and uses a second pix2pixHD-style generator, augmented with a temporal smoothing loss, to produce the target video. The authors report experiments on two videos of one target individual, driven by two different source individuals, with only qualitative results shown in Fig. 4. The central claim is that the synthesized target sequences are photorealistic and consistent with the source in body motion, expression, and pose.

Significance. If the method worked as claimed, it would provide a practical way to combine body motion transfer with face reenactment in a single pipeline, using FAUP as an identity-independent intermediate representation to avoid explicit facial landmark normalization. The high-level idea is clearly stated and the pipeline is easy to follow, and the authors are explicit about their use of existing building blocks such as pix2pixHD and OpenFace. However, the significance cannot currently be assessed because the paper provides no quantitative metrics, no comparisons with prior work, no ablations, no failure analysis, no dataset details, and no code. The only evidence of effectiveness is two unquantified example videos. The contribution is therefore unverified as presented.

major comments (3)
  1. [Section 6] The paper's central claim of effectiveness rests entirely on two qualitative examples. Section 6 reports only that two videos of one individual were synthesized and that the results 'are photorealistic and consistent with the source sequence,' with Fig. 4 as the sole evidence. There are no quantitative metrics (e.g., FID, LPIPS, user study, keypoint accuracy, landmark distance), no baselines, no ablations, and no failure cases. Because the entire contribution depends on the demonstrated quality of the synthesized videos, this lack of evaluation is load-bearing and prevents verification of the central claim.
  2. [Section 4] The FAUP-to-landmark generator is trained only on target-person (FAUP, facial landmark) pairs, yet at test time it is fed source-person FAUP values ('During the transferring phase, we directly use source's FAUP to synthesize target's facial landmark'). The paper provides no evidence that OpenFace FAUP values are sufficiently identity-invariant or that source FAUP values lie within the target's training distribution. If source FAUP falls outside that distribution, the generated landmarks may not preserve the source expression, and stage 3's generator, trained on target UBKP-FL/video pairs, will receive out-of-distribution input. No domain adaptation, augmentation, or out-of-distribution analysis is provided, so the claimed expression consistency rests on an unvalidated distribution-shift assumption.
  3. [Equations (1)-(4)] The loss formulations contain notation inconsistencies that make the exact training objective ambiguous. Eq. (1) defines the per-scale adversarial loss LGAN(G,D_k), but Eq. (2) writes LGAN(G,D) without specifying the scale or how the three discriminators are combined. Eq. (3) contains a typo 'xx−L' in the first expectation and defines LtS(G,D) with a single discriminator D, while Eq. (4) sums LtS over k=1,2,3, presumably over D_k, but this is never made explicit. Since these losses define the training procedure, this ambiguity affects reproducibility.
minor comments (4)
  1. [Abstract] The abstract states 'Experimental results demonstrated the effectiveness of our method' before any metrics are presented; consider tempering this claim or making the evaluation explicit.
  2. [Section 5] The description of the temporal setup would benefit from a precise definition of the notation: the text says the first output G(x_{t-L}) is conditioned on a zero image z, but the equations do not include z or explain how the zero image is used.
  3. [Fig. 2 and Fig. 3 captions] The captions use inconsistent notation with the main text: Fig. 2 refers to x_t and y_t without defining the subscript, and Fig. 3 says 'The discriminator D attempts...' while Eq. (4) uses D_1, D_2, D_3.
  4. [Section 2] There is a capitalization error in 'Different from above works, We propose...' where 'We' should be lowercase.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is a supervised train/test procedure with external building blocks; the weak evaluation is an evidence issue, not a circularity issue.

full rationale

The paper's derivation chain is a standard supervised generation pipeline. Stage 2 trains generator G on the target person's own (FAUP, facial-landmark) pairs with adversarial and L1 losses (Eqs. 1-2), then at transfer time feeds source FAUP into G to synthesize target landmarks. The predicted target landmarks are not used to define the training objective; the objective compares G(x_t) with ground-truth target landmarks y_t. Stage 3 similarly trains pix2pixHD with temporal smoothing on target (UBKP-FL, video) pairs and then tests on UBKP-FL images built from the source video. No fitted parameter is later renamed as a prediction, and no output quantity is an input by construction. The FAUP representation is adopted from OpenFace [3] and the UBKP normalization and temporal-smoothing idea from Everybody Dance Now [6]; these are external prior works, not self-citations by the present authors, so no self-citation chain is load-bearing. The paper's claims that FAUP is identity-independent and that the two qualitative examples demonstrate the method are empirically weak assumptions and thin evidence, but those are correctness/robustness concerns, not circularity. The manuscript itself notes only an efficiency limitation in Section 7, which is unrelated to circularity. Therefore the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its central claim rests on four domain assumptions about representation invariance, pose normalization, upstream estimators, and data availability, plus several unspecified loss-weight hyperparameters that would need tuning to reproduce results.

free parameters (5)
  • lambda (L1 loss weight) = not specified
    Chosen by hand in Eq. 2 to balance adversarial and L1 losses; value not reported.
  • alpha (feature matching loss weight) = not specified
    Chosen by hand in Eq. 4; value not reported.
  • beta (VGG perceptual loss weight) = not specified
    Chosen by hand in Eq. 4; value not reported.
  • L (temporal window length) = not specified
    Number of consecutive frames considered in temporal smoothing loss; value not reported.
  • number of discriminator scales = 3
    Fixed at 3 pix2pixHD discriminators (k=1,2,3).
assumptions (4)
  • domain assumption FAUP is identity-independent and captures facial expressions without spatial normalization.
    Stated in Section 1 to justify skipping landmark normalization; if false, cross-person expression transfer fails.
  • domain assumption Global pose normalization from Everybody Dance Now transfers upper-body keypoints between arbitrary source and target body shapes.
    Used in Section 3 to align source UBKP to target proportions; assumes skeletal proportions scale linearly.
  • domain assumption Pretrained pose detector and OpenFace produce accurate UBKP and FAUP estimates on both source and target videos.
    Stage 1 relies on these estimates; estimation errors propagate into the synthesized landmarks and frames.
  • domain assumption A target-person video with paired FAUP and facial landmark observations is available for training.
    Section 4 trains the FAUP-to-landmark generator on target's own pairs; without such data the pipeline cannot be applied to a new target.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video synthesis of human upper body with realistic face." pith.science (2026). https://pith.science/paper/HPA3OJ47

@misc{pith2026190806607,
  author       = {Pith},
  title        = {Pith review of: Video synthesis of human upper body with realistic face},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPA3OJ47}},
  note         = {Machine review of arXiv:1908.06607}
}
read the original abstract

This paper presents a generative adversarial learning-based human upper body video synthesis approach to generate an upper body video of target person that is consistent with the body motion, face expression, and pose of the person in source video. We use upper body keypoints, facial action units and poses as intermediate representations between source video and target video. Instead of directly transferring the source video to the target video, we firstly map the source person's facial action units and poses into the target person's facial landmarks, then combine the normalized upper body keypoints and generated facial landmarks with spatio-temporal smoothing to generate the corresponding target video's image. Experimental results demonstrated the effectiveness of our method.

Figures

Figures reproduced from arXiv: 1908.06607 by the authors.

Figure 1
Figure 1. The pipeline of transferring motion from source to target. Stage1: UBKP estimation and FAUP extraction from source video. Stage 2: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. shows the training setup. Since the extracted FAUP does not have coordinate information like UBKP, we fill the FAUP vector into the center of an empty image and then feed it into our network for training, in order to facilitate loading data in the training phase. This also allows our network to have spatial coordinate constraints during training, which can accelerate the convergence. The objective function is given … view at source ↗
Figure 3
Figure 3. We use pair (xt ,(G(xt−L),...,G(xt−1))) to learn the mapping G to generate G(xt). VGG perceptual reconstruction loss and feature matching loss LFM are used to reduce the gap between G(xt) and ground truth yt .The discriminator D attempts to differentiate the real temporally coherent sequence from the fake sequence. Section 1 Section 2 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Transfer results. In each section, the first row shows [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Realistic Face-to-Face Conversation System based on Deep Neural Networks

    cs.CV 2019-08 conditional novelty 4.0 of 10

    A conversation avatar that maps a speaker's facial actions and head pose to a listener's reactions, maps text to speaking expressions, and renders the results as realistic face images using Pix2PixHD.

  2. A Neural Virtual Anchor Synthesizer based on Seq2Seq and GAN Models

    cs.CV 2019-08 reject novelty 4.0 of 10

    A virtual anchor face video is synthesized from text by predicting action units and head poses with a Seq2Seq model and rendering frames with a Pix2PixHD generator, as shown in qualitative examples.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    W. J. Baddar, G. Gu, S. Lee, and Y . M. Ro. Dynamics transfer gan: Generating video by transferring arbitrary temporal dynamics from a source video to a single target image. arXiv preprint arXiv:1712.03534, 2017

    work page Pith review arXiv 2017

  2. [2]

    Balakrishnan, A

    G. Balakrishnan, A. Zhao, A. V . Dalca, F. Durand, and J. Guttag. Synthesizing images of humans in unseen poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 8340–8348, 2018

    work page 2018

  3. [3]

    Baltruˇsaitis, P

    T. Baltruˇsaitis, P. Robinson, and L.-P. Morency. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Confer- ence on Applications of Computer Vision (WACV) , pp. 1–10. IEEE, 2016

    2016

  4. [4]

    Bansal, S

    A. Bansal, S. Ma, D. Ramanan, and Y . Sheikh. Recycle-gan: Unsuper- vised video retargeting. european conference on computer vision, pp. 122–138, 2018

    2018

  5. [5]

    Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 7291–7299, 2017

    work page 2017

  6. [6]

    C. Chan, S. Ginosar, T. Zhou, and A. A. Efros. Everybody dance now. arXiv: Graphics, 2018

    2018

  7. [7]

    Esser and E

    P. Esser and E. Sutter. A variational u-net for conditional appearance and shape generation. computer vision and pattern recognition , pp. 8857–8866, 2018

    work page 2018

  8. [8]

    Goodfellow, J

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672–2680, 2014

    2014

Show all 20 references
  1. [9]

    Isola, J

    P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. computer vision and pattern recognition, pp. 5967–5976, 2017

    2017

  2. [10]

    X. Jin, Y . Qi, and S. Wu. Cyclegan face-off. arXiv preprint arXiv:1712.03451, 2017

    2017 arXiv

  3. [11]

    D. Joo, D. Kim, and J. Kim. Generating a fusion image: One’s identity and another’s shape. computer vision and pattern recognition , pp. 1635–1643, 2018

    2018

  4. [12]

    L. Ma, J. Xu, Q. Sun, B. Schiele, T. Tuytelaars, and L. V . Gool. Pose guided person image generation. 2017

    2017

  5. [13]

    Simon, H

    T. Simon, H. Joo, I. Matthews, and Y . Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1145–1153, 2017

    2017

  6. [14]

    Tulyakov, M.-Y

    S. Tulyakov, M.-Y . Liu, X. Yang, and J. Kautz. Mocogan: Decom- posing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1526–1535, 2018

    2018

  7. [15]

    Villegas, J

    R. Villegas, J. Yang, Y . Zou, S. Sohn, X. Lin, and H. Lee. Learning to generate long-term future via hierarchical prediction. international conference on machine learning, pp. 3560–3569, 2017

    2017

  8. [16]

    T. Wang, M. Liu, J. Zhu, G. Guilin, A. J. Tao, J. Kautz, and B. Catan- zaro. Video-to-video synthesis. neural information processing systems, pp. 1144–1156, 2018

    2018

  9. [17]

    T. Wang, M. Liu, J. Zhu, A. J. Tao, J. Kautz, and B. Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. computer vision and pattern recognition , pp. 8798–8807, 2018

    2018

  10. [18]

    W. Wu, Y . Zhang, C. Li, C. Qian, and C. C. Loy. Reenactgan: Learn- ing to reenact faces via boundary transfer. european conference on computer vision, pp. 622–638, 2018

    2018

  11. [19]

    R. Xu, Z. Zhou, W. Zhang, and Y . Yu. Face transfer with generative adversarial network. arXiv preprint arXiv:1710.06090, 2017

    2017 arXiv

  12. [20]

    J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. international conference on computer vision, pp. 2242–2251, 2017

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.